hub (unreleased): two end dates warn the operator 14 days before — an operator token and a box's web certificate (R-1379, D77); the other sixteen as a design table

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-14 18:14:23 +01:00
1 parent 68292e40f6
commit 86bb912a4b
11 files changed
+527 -5

No files matched your search

@@ -374,6 +374,8 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
| `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` |
| `box_clock_wrong` | warning | **[built, not deployed — hub `main` after v0.181.0; needs an agent that stamps `sent_at`, built and not released]** the box's clock at the moment of sending is more than **one minute** from the hub's clock at arrival, in either direction (D79, `09` §3 decision 300); told once per direction. The hub cannot tell whose clock is wrong: a second box off in the same direction makes the alarm say „check the hub's own clock first”. An agent without `sent_at` is never judged | the first report within **30 seconds** → `box_clock_ok` (info) | `api/box_clock_test.go`; agent `hub/client_sentat_test.go` |
| `operator_token_ends_soon` | warning; error when the date passed | **[built, not deployed — hub `main` after v0.181.0; D77, `09` §3 decision 298, §12 below]** an operator access token (hub §17) that is not revoked ends within **14 days**; checked hourly; once per token, and once more when the date passes with the token still there | none (a new token is a new token) | `monitor/end_dates_test.go` |
| `web_cert_ends_soon` | warning; error when the date passed | **[built, not deployed — hub `main` after v0.181.0; D77, `09` §3 decision 298, §12 below]** needs a controller after 0.352.1 (built, not released): its report's `web_certs` list names a web certificate that ends within **14 days** — the box renews 30 days before the end, so this means the renewal has failed for about two weeks; once per (household, name, end date) | none (a renewed certificate has a new end date) | `monitor/end_dates_test.go`; controller `infra/webcert_test.go` |
| `hub_daily_job_skipped` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: the newest slot of `deadline-check` or `offsite-key-audit` passed while the hub was not running; once per missed slot | none (the next slot runs) | `monitor/daily_ledger_test.go`, `cmd/hub/r1383_wiring_test.go` |
| `operator_mail_lost` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: operator mails of the last 48 h whose newest delivery record is `failed` — one mail for all, with each lost mail's own sentence; and a row (no mail) when three retries gave up | none | `notify/lost_operator_mail_test.go` |
| `signed_job_not_taken` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** a signed job is still in a box's queue after two hours; once per job | none (the job leaves the queue when the box takes it) | `monitor/signed_job_waits_test.go` |
@@ -772,3 +774,39 @@ only a word (`WORD_ONLY`: „security announcements", „no longer receives secu
another program in the repo (`PACKAGE_NOT_OURS`: Gitea's runner, Home Assistant's phone apps, Uptime Kuma's rootless image) is
not listed; a name not in the table is listed — the safe direction. A `YY.MM.YYYYMMDD` tag is not „newer". Test W16.
## 12. End dates — who knows the date, and who is warned [R-1379, D77, `09` §3 decision 298]
The End-dates list (`audits/end-dates-list-2026-11-11/`) found 18 end dates with no warning before them. The ruling:
the product warns 14 days before the two it can read itself; the rest stays a list the operator reads.
### 12.1 The two that are built `[FACT — built, not released: hub `main` after v0.181.0 and controller `main` after 0.352.1, 2026-11-14]`
| End date | Who knows the date | How it reaches the hub | The warning |
|---|---|---|---|
| A box's web certificate | **only the box.** Traefik in the guest asks Let's Encrypt and keeps the certificate in its own store (`<stacks>/traefik/acme.json`). The household's public names end at Cloudflare, so nobody outside can read it | the controller reads the store's certificate fields — names and end dates, never a key (`infra.ReadWebCerts`; the store's shape was measured on three boxes on 2026-11-11) — and sends `web_certs` in its report | `web_cert_ends_soon`, 14 days before; an error if the date passes |
| An operator access token | the hub (`api_tokens.expires_at`) | — | `operator_token_ends_soon`, 14 days before; an error if the date passes |
What it does not do: a box with NO certificate in the store (no ACME e-mail: Traefik serves its own certificate) is not
told — the field is absent. A box that is switched off is judged on its last report: the warning is still true (an off
box cannot renew). A token or certificate that ended more than 14 days ago raises nothing at the first run (no news).
**Not proven:** on a box or a deployed hub; a real failed renewal has never been seen.
### 12.2 The other sixteen `[DESIGN — proposed, not built]`
| End date (block) | Who knows the date today | Could the product read it? | Proposed |
|---|---|---|---|
| The hub address's certificate, and the download, ISO, Gitea and website names (DH20) | anybody: readable from outside with no login | yes — the hub can open a TLS connection to its own five names once a day | **build next:** the same checker, a third source. Small. Not built tonight: it is a call from the hub to outside names, and the name list needs the operator's yes |
| A node's Tailscale key (DH30) | the Tailscale admin page; one date is in `operations/tailscale.md` | not from a box (the service does not run on two of three) | the operator's list |
| The package signing keys (DB16) | measured: the soonest key in use ends 2029-01-15 | yes, by the root wrapper's facts read | no warning now; read again in the yearly end-dates pass |
| The 30-day restore copy (DB24) | the box (the copy's own date) | yes | not an alarm: a page fact. The household's restore page shows the date already; nothing for the operator |
| The end of security fixes for Debian 13 and Proxmox 9 (DB17) | nobody in this project: no document gives the day | no — it is a published date, not a machine fact | the operator's list, with the two dates written into `11-os-updates.md` once they are read |
| App images and the images the controller starts (DB18, DB19) | no end date exists; the end is the day a registry stops serving | no | none; the nightly security watch covers the versions |
| The agent's local-API certificate (DB03) | source: 10 years from the install | yes (the agent made it) | no warning before 2035; a row then |
| A household's own Cloudflare token (DH23) | only the household (it set the end at Cloudflare) | **maybe:** the hub's zone-reach check calls Cloudflare's zone list (`cloudflare/reach.go`), which gives no end date. Cloudflare has a token-verify call that is said to give one (`expires_on`) — NOT read in this project, NOT seen in an answer | **design question, with a spike first:** one read-only token-verify call on a test household's token, to see the field. Not done tonight: a call to Cloudflare is forbidden in this run |
| The mail service key; domains and registrars; the Cloudflare account, the site publish token and the ISO store keys; the Hetzner account; Gitea and registry logins; the tokens in CC's secret store (DH24, DH27, DH28, DH29, DH31, DH32) | the account's own page; no document holds a date | no — each needs a login the product must not hold | **the operator's list:** one table of name, where to read the date, the date, read on. It does not exist yet (R-1381 asks the operator to read the dates only they can read) |
**A third source that is small and was left out on purpose:** the hub's own five public names (first row). It needs
no secret, and its cost is one outbound connection a day; it waits for a yes because tonight's brief forbids outside
calls and the list of names is a promise about what the hub watches.
@@ -0,0 +1,44 @@
== MUTATION: more than a name leaves the box (a slice of the stored certificate field is appended to the name)
RED on the mutant, GREEN on the real file
--- FAIL: TestReadWebCerts_EndDatesAndNoKeyMaterial (0.00s)
webcert_test.go:68: certificates = [{Domain:old.example.hu Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K NotAfter:2026-11-20T01:46:00Z Issuer:old.example.hu} {Domain:*.example.hu cFFzCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K NotAfter:2027-0
== MUTATION: the end date is the start date
RED on the mutant, GREEN on the real file
--- FAIL: TestReadWebCerts_EndDatesAndNoKeyMaterial (0.00s)
webcert_test.go:68: certificates = [{Domain:old.example.hu NotAfter:2026-08-22T01:46:00Z Issuer:old.example.hu} {Domain:*.example.hu NotAfter:2026-10-21T01:46:00Z Issuer:*.example.hu}], want old.example.hu 2026-11-20 first, then *
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.01s)
webcert_test.go:117: cap: 8 entries, first {Domain:na.example.hu NotAfter:2026-10-03T00:00:00Z Issuer:na.example.hu}
== MUTATION: the soonest end is not first
RED on the mutant, GREEN on the real file
--- FAIL: TestReadWebCerts_EndDatesAndNoKeyMaterial (0.00s)
webcert_test.go:68: certificates = [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu} {Domain:old.example.hu NotAfter:2026-11-20T01:46:00Z Issuer:old.example.hu}], want old.example.hu 2026-11-20 first, then *
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.00s)
webcert_test.go:117: cap: 8 entries, first {Domain:nl.example.hu NotAfter:2027-01-12T00:00:00Z Issuer:nl.example.hu}
== MUTATION: a broken store's content is put in the error
RED on the mutant, GREEN on the real file
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.01s)
webcert_test.go:102: a broken store: err = web certificate store is not readable JSON: {"letsencrypt": SECRET-LOOKING-BYTES, want an error that does not quote the file
== MUTATION: an empty store is an error (every new box would warn)
RED on the mutant, GREEN on the real file
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.00s)
webcert_test.go:93: an empty store: [], web certificate store is not readable JSON
== MUTATION: no cap on the list
RED on the mutant, GREEN on the real file
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.01s)
webcert_test.go:117: cap: 12 entries, first {Domain:na.example.hu NotAfter:2027-01-01T00:00:00Z Issuer:na.example.hu}
6 mutations, 0 not a red-proof
== MUTATION: the report reads another folder than Traefik's
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
== MUTATION: BuildReport never fills the field
RED on the mutant, GREEN on the real file
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
web_certs_report_test.go:57: BuildReport assigns r.WebCerts from webCertsForReport 0 time(s), want 1
2 mutations, 1 not a red-proof
== MUTATION: the report reads another folder than Traefik's — done again: the first run was GREEN (the test had no certificate in the folder; one was added, with a decoy in the wrong folder)
RED on the mutant, GREEN on the real file
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
web_certs_report_test.go:50: certificates read for the report = [{Domain:decoy.example.hu NotAfter:2026-12-01T00:00:00Z Issuer:decoy.example.hu}], want the one in <stacks>/traefik/acme.json
1 mutations, 0 not a red-proof
@@ -0,0 +1,60 @@
== MUTATION: nothing is told before the end date (the old behaviour)
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
end_dates_test.go:65: inside the 14 days, 31 hourly checks: 0 event(s), want ONE per end date: []
== MUTATION: the warning comes at 30 days, not 14
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
end_dates_test.go:58: at more than 14 days: [|operator_token_ends_soon|warning|The operator access token „CC hub token” ends in 16 day(s), on 2026-11-30. On that day every script that uses it gets 401 (for one: the release step th
== MUTATION: the warning is repeated at every hourly check
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
end_dates_test.go:65: inside the 14 days, 31 hourly checks: 62 event(s), want ONE per end date: [|operator_token_ends_soon|warning|The operator access token „CC hub token” ends in 13 day(s), on 2026-11-30. On that day every script
== MUTATION: the passed end date is not told again as ended
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
end_dates_test.go:82: after the end date: []
== MUTATION: a renewed certificate is still told (the key ignores the end date)
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_RenewedAndLongGoneAreQuiet (0.00s)
end_dates_test.go:112: the new certificate's own warning: 1 event(s)
== MUTATION: a token that ended long ago is told at the first run
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_RenewedAndLongGoneAreQuiet (0.00s)
end_dates_test.go:97: a token that ended 120 days ago and a certificate 40 days out: [|operator_token_ends_soon|error|The operator access token „old” ENDED on 2026-07-17 and is not replaced: every script that uses it gets 401. Do:
== MUTATION: the box's name is not checked
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/monitor [build failed]
== MUTATION: the box's list has no cap
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
end_dates_test.go:141: household „many”: 20 event(s), want the cap 8
== MUTATION: the token's name is not checked
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
end_dates_test.go:135: an unchecked word of a box or a token name reached an event:
== MUTATION: a duplicate row of a household is read twice
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
end_dates_test.go:135: an unchecked word of a box or a token name reached an event:
10 mutations, 1 not a red-proof
(the wiring mutation did not run: its spec matched nothing; done again at the end)
== MUTATION: the end-date warnings reach the household
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDateEventsAreOperatorOnly (0.00s)
box_clock_operator_only_test.go:18: web_cert_ends_soon must be operator-only
1 mutations, 0 not a red-proof
== MUTATION: the box's name is not checked — done again: the first mutant did not compile (an import was left unused)
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
end_dates_test.go:135: an unchecked word of a box or a token name reached an event:
1 mutations, 0 not a red-proof
== MUTATION: the end-date check is not on the sweep — done again: gofmt had moved the comment, so the first spec matched nothing
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
r1383_wiring_test.go:72: main.go: end-date sweep — found 0 time(s), want 1
1 mutations, 0 not a red-proof
+1 -1
View File
@@ -331,7 +331,7 @@ stopping line that lies.
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | NARROWED — the two ruled warnings are built, not released. **2026-11-14 (night build):** the hub warns the operator 14 days before an operator token ends and 14 days before a box's web certificate ends (the box reads the date from its own store and sends it in its report) — `08-alarm-ladder.md` §12.1. The other sixteen are a design table there (§12.2): one is small and waits for a yes (the hub's own five public names), one is a design question (a household's Cloudflare token), the rest stay the operator's list. Not seen on a box or a deployed hub. OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — **2026-11-14 (night build, second part):** the two things release 2a did not tell are built, not released: the System page has a cell for the kernel clean-up (removed, kept, space gained) and one for the security-profile refresh (the kernel, how many profiles changed, any failed service it restarted) — `11` §5.11.1, §5.12. On the way a display defect was fixed: a clean-up hid a kernel that fell back the night before in the „Kernel step” cell. NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |