hub (unreleased): two end dates warn the operator 14 days before — an operator token and a box's web certificate (R-1379, D77); the other sixteen as a design table
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
68292e40f6
commit
86bb912a4b
11 files changed
+527
-5
No files matched your search
@@ -374,6 +374,8 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
||||
| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
|
||||
| `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` |
|
||||
| `box_clock_wrong` | warning | **[built, not deployed — hub `main` after v0.181.0; needs an agent that stamps `sent_at`, built and not released]** the box's clock at the moment of sending is more than **one minute** from the hub's clock at arrival, in either direction (D79, `09` §3 decision 300); told once per direction. The hub cannot tell whose clock is wrong: a second box off in the same direction makes the alarm say „check the hub's own clock first”. An agent without `sent_at` is never judged | the first report within **30 seconds** → `box_clock_ok` (info) | `api/box_clock_test.go`; agent `hub/client_sentat_test.go` |
|
||||
| `operator_token_ends_soon` | warning; error when the date passed | **[built, not deployed — hub `main` after v0.181.0; D77, `09` §3 decision 298, §12 below]** an operator access token (hub §17) that is not revoked ends within **14 days**; checked hourly; once per token, and once more when the date passes with the token still there | none (a new token is a new token) | `monitor/end_dates_test.go` |
|
||||
| `web_cert_ends_soon` | warning; error when the date passed | **[built, not deployed — hub `main` after v0.181.0; D77, `09` §3 decision 298, §12 below]** needs a controller after 0.352.1 (built, not released): its report's `web_certs` list names a web certificate that ends within **14 days** — the box renews 30 days before the end, so this means the renewal has failed for about two weeks; once per (household, name, end date) | none (a renewed certificate has a new end date) | `monitor/end_dates_test.go`; controller `infra/webcert_test.go` |
|
||||
| `hub_daily_job_skipped` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: the newest slot of `deadline-check` or `offsite-key-audit` passed while the hub was not running; once per missed slot | none (the next slot runs) | `monitor/daily_ledger_test.go`, `cmd/hub/r1383_wiring_test.go` |
|
||||
| `operator_mail_lost` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: operator mails of the last 48 h whose newest delivery record is `failed` — one mail for all, with each lost mail's own sentence; and a row (no mail) when three retries gave up | none | `notify/lost_operator_mail_test.go` |
|
||||
| `signed_job_not_taken` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** a signed job is still in a box's queue after two hours; once per job | none (the job leaves the queue when the box takes it) | `monitor/signed_job_waits_test.go` |
|
||||
@@ -772,3 +774,39 @@ only a word (`WORD_ONLY`: „security announcements", „no longer receives secu
|
||||
another program in the repo (`PACKAGE_NOT_OURS`: Gitea's runner, Home Assistant's phone apps, Uptime Kuma's rootless image) is
|
||||
not listed; a name not in the table is listed — the safe direction. A `YY.MM.YYYYMMDD` tag is not „newer". Test W16.
|
||||
|
||||
|
||||
|
||||
## 12. End dates — who knows the date, and who is warned [R-1379, D77, `09` §3 decision 298]
|
||||
|
||||
The End-dates list (`audits/end-dates-list-2026-11-11/`) found 18 end dates with no warning before them. The ruling:
|
||||
the product warns 14 days before the two it can read itself; the rest stays a list the operator reads.
|
||||
|
||||
### 12.1 The two that are built `[FACT — built, not released: hub `main` after v0.181.0 and controller `main` after 0.352.1, 2026-11-14]`
|
||||
|
||||
| End date | Who knows the date | How it reaches the hub | The warning |
|
||||
|---|---|---|---|
|
||||
| A box's web certificate | **only the box.** Traefik in the guest asks Let's Encrypt and keeps the certificate in its own store (`<stacks>/traefik/acme.json`). The household's public names end at Cloudflare, so nobody outside can read it | the controller reads the store's certificate fields — names and end dates, never a key (`infra.ReadWebCerts`; the store's shape was measured on three boxes on 2026-11-11) — and sends `web_certs` in its report | `web_cert_ends_soon`, 14 days before; an error if the date passes |
|
||||
| An operator access token | the hub (`api_tokens.expires_at`) | — | `operator_token_ends_soon`, 14 days before; an error if the date passes |
|
||||
|
||||
What it does not do: a box with NO certificate in the store (no ACME e-mail: Traefik serves its own certificate) is not
|
||||
told — the field is absent. A box that is switched off is judged on its last report: the warning is still true (an off
|
||||
box cannot renew). A token or certificate that ended more than 14 days ago raises nothing at the first run (no news).
|
||||
**Not proven:** on a box or a deployed hub; a real failed renewal has never been seen.
|
||||
|
||||
### 12.2 The other sixteen `[DESIGN — proposed, not built]`
|
||||
|
||||
| End date (block) | Who knows the date today | Could the product read it? | Proposed |
|
||||
|---|---|---|---|
|
||||
| The hub address's certificate, and the download, ISO, Gitea and website names (DH20) | anybody: readable from outside with no login | yes — the hub can open a TLS connection to its own five names once a day | **build next:** the same checker, a third source. Small. Not built tonight: it is a call from the hub to outside names, and the name list needs the operator's yes |
|
||||
| A node's Tailscale key (DH30) | the Tailscale admin page; one date is in `operations/tailscale.md` | not from a box (the service does not run on two of three) | the operator's list |
|
||||
| The package signing keys (DB16) | measured: the soonest key in use ends 2029-01-15 | yes, by the root wrapper's facts read | no warning now; read again in the yearly end-dates pass |
|
||||
| The 30-day restore copy (DB24) | the box (the copy's own date) | yes | not an alarm: a page fact. The household's restore page shows the date already; nothing for the operator |
|
||||
| The end of security fixes for Debian 13 and Proxmox 9 (DB17) | nobody in this project: no document gives the day | no — it is a published date, not a machine fact | the operator's list, with the two dates written into `11-os-updates.md` once they are read |
|
||||
| App images and the images the controller starts (DB18, DB19) | no end date exists; the end is the day a registry stops serving | no | none; the nightly security watch covers the versions |
|
||||
| The agent's local-API certificate (DB03) | source: 10 years from the install | yes (the agent made it) | no warning before 2035; a row then |
|
||||
| A household's own Cloudflare token (DH23) | only the household (it set the end at Cloudflare) | **maybe:** the hub's zone-reach check calls Cloudflare's zone list (`cloudflare/reach.go`), which gives no end date. Cloudflare has a token-verify call that is said to give one (`expires_on`) — NOT read in this project, NOT seen in an answer | **design question, with a spike first:** one read-only token-verify call on a test household's token, to see the field. Not done tonight: a call to Cloudflare is forbidden in this run |
|
||||
| The mail service key; domains and registrars; the Cloudflare account, the site publish token and the ISO store keys; the Hetzner account; Gitea and registry logins; the tokens in CC's secret store (DH24, DH27, DH28, DH29, DH31, DH32) | the account's own page; no document holds a date | no — each needs a login the product must not hold | **the operator's list:** one table of name, where to read the date, the date, read on. It does not exist yet (R-1381 asks the operator to read the dates only they can read) |
|
||||
|
||||
**A third source that is small and was left out on purpose:** the hub's own five public names (first row). It needs
|
||||
no secret, and its cost is one outbound connection a day; it waits for a yes because tonight's brief forbids outside
|
||||
calls and the list of names is a promise about what the hub watches.
|
||||
@@ -0,0 +1,44 @@
|
||||
== MUTATION: more than a name leaves the box (a slice of the stored certificate field is appended to the name)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReadWebCerts_EndDatesAndNoKeyMaterial (0.00s)
|
||||
webcert_test.go:68: certificates = [{Domain:old.example.hu Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K NotAfter:2026-11-20T01:46:00Z Issuer:old.example.hu} {Domain:*.example.hu cFFzCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K NotAfter:2027-0
|
||||
== MUTATION: the end date is the start date
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReadWebCerts_EndDatesAndNoKeyMaterial (0.00s)
|
||||
webcert_test.go:68: certificates = [{Domain:old.example.hu NotAfter:2026-08-22T01:46:00Z Issuer:old.example.hu} {Domain:*.example.hu NotAfter:2026-10-21T01:46:00Z Issuer:*.example.hu}], want old.example.hu 2026-11-20 first, then *
|
||||
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.01s)
|
||||
webcert_test.go:117: cap: 8 entries, first {Domain:na.example.hu NotAfter:2026-10-03T00:00:00Z Issuer:na.example.hu}
|
||||
== MUTATION: the soonest end is not first
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReadWebCerts_EndDatesAndNoKeyMaterial (0.00s)
|
||||
webcert_test.go:68: certificates = [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu} {Domain:old.example.hu NotAfter:2026-11-20T01:46:00Z Issuer:old.example.hu}], want old.example.hu 2026-11-20 first, then *
|
||||
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.00s)
|
||||
webcert_test.go:117: cap: 8 entries, first {Domain:nl.example.hu NotAfter:2027-01-12T00:00:00Z Issuer:nl.example.hu}
|
||||
== MUTATION: a broken store's content is put in the error
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.01s)
|
||||
webcert_test.go:102: a broken store: err = web certificate store is not readable JSON: {"letsencrypt": SECRET-LOOKING-BYTES, want an error that does not quote the file
|
||||
== MUTATION: an empty store is an error (every new box would warn)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.00s)
|
||||
webcert_test.go:93: an empty store: [], web certificate store is not readable JSON
|
||||
== MUTATION: no cap on the list
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReadWebCerts_EmptyMissingAndBroken (0.01s)
|
||||
webcert_test.go:117: cap: 12 entries, first {Domain:na.example.hu NotAfter:2027-01-01T00:00:00Z Issuer:na.example.hu}
|
||||
|
||||
6 mutations, 0 not a red-proof
|
||||
== MUTATION: the report reads another folder than Traefik's
|
||||
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
|
||||
== MUTATION: BuildReport never fills the field
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
|
||||
web_certs_report_test.go:57: BuildReport assigns r.WebCerts from webCertsForReport 0 time(s), want 1
|
||||
|
||||
2 mutations, 1 not a red-proof
|
||||
== MUTATION: the report reads another folder than Traefik's — done again: the first run was GREEN (the test had no certificate in the folder; one was added, with a decoy in the wrong folder)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
|
||||
web_certs_report_test.go:50: certificates read for the report = [{Domain:decoy.example.hu NotAfter:2026-12-01T00:00:00Z Issuer:decoy.example.hu}], want the one in <stacks>/traefik/acme.json
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,60 @@
|
||||
== MUTATION: nothing is told before the end date (the old behaviour)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
|
||||
end_dates_test.go:65: inside the 14 days, 31 hourly checks: 0 event(s), want ONE per end date: []
|
||||
== MUTATION: the warning comes at 30 days, not 14
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
|
||||
end_dates_test.go:58: at more than 14 days: [|operator_token_ends_soon|warning|The operator access token „CC hub token” ends in 16 day(s), on 2026-11-30. On that day every script that uses it gets 401 (for one: the release step th
|
||||
== MUTATION: the warning is repeated at every hourly check
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
|
||||
end_dates_test.go:65: inside the 14 days, 31 hourly checks: 62 event(s), want ONE per end date: [|operator_token_ends_soon|warning|The operator access token „CC hub token” ends in 13 day(s), on 2026-11-30. On that day every script
|
||||
== MUTATION: the passed end date is not told again as ended
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FourteenDaysOnceThenEndedOnce (0.00s)
|
||||
end_dates_test.go:82: after the end date: []
|
||||
== MUTATION: a renewed certificate is still told (the key ignores the end date)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_RenewedAndLongGoneAreQuiet (0.00s)
|
||||
end_dates_test.go:112: the new certificate's own warning: 1 event(s)
|
||||
== MUTATION: a token that ended long ago is told at the first run
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_RenewedAndLongGoneAreQuiet (0.00s)
|
||||
end_dates_test.go:97: a token that ended 120 days ago and a certificate 40 days out: [|operator_token_ends_soon|error|The operator access token „old” ENDED on 2026-07-17 and is not replaced: every script that uses it gets 401. Do:
|
||||
== MUTATION: the box's name is not checked
|
||||
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/monitor [build failed]
|
||||
== MUTATION: the box's list has no cap
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
|
||||
end_dates_test.go:141: household „many”: 20 event(s), want the cap 8
|
||||
== MUTATION: the token's name is not checked
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
|
||||
end_dates_test.go:135: an unchecked word of a box or a token name reached an event:
|
||||
== MUTATION: a duplicate row of a household is read twice
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
|
||||
end_dates_test.go:135: an unchecked word of a box or a token name reached an event:
|
||||
|
||||
10 mutations, 1 not a red-proof
|
||||
(the wiring mutation did not run: its spec matched nothing; done again at the end)
|
||||
== MUTATION: the end-date warnings reach the household
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDateEventsAreOperatorOnly (0.00s)
|
||||
box_clock_operator_only_test.go:18: web_cert_ends_soon must be operator-only
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: the box's name is not checked — done again: the first mutant did not compile (an import was left unused)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_TheBoxesWordsAreChecked (0.00s)
|
||||
end_dates_test.go:135: an unchecked word of a box or a token name reached an event:
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: the end-date check is not on the sweep — done again: gofmt had moved the comment, so the first spec matched nothing
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
|
||||
r1383_wiring_test.go:72: main.go: end-date sweep — found 0 time(s), want 1
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
@@ -331,7 +331,7 @@ stopping line that lies.
|
||||
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
|
||||
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
|
||||
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | NARROWED — the two ruled warnings are built, not released. **2026-11-14 (night build):** the hub warns the operator 14 days before an operator token ends and 14 days before a box's web certificate ends (the box reads the date from its own store and sends it in its report) — `08-alarm-ladder.md` §12.1. The other sixteen are a design table there (§12.2): one is small and waits for a yes (the hub's own five public names), one is a design question (a household's Cloudflare token), the rest stay the operator's list. Not seen on a box or a deployed hub. OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
|
||||
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — **2026-11-14 (night build, second part):** the two things release 2a did not tell are built, not released: the System page has a cell for the kernel clean-up (removed, kept, space gained) and one for the security-profile refresh (the kernel, how many profiles changed, any failed service it restarted) — `11` §5.11.1, §5.12. On the way a display defect was fixed: a clean-up hid a kernel that fell back the night before in the „Kernel step” cell. NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
|
||||
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
|
||||
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
|
||||
|
||||
@@ -3,6 +3,14 @@
|
||||
- `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at
|
||||
every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change.
|
||||
- **Release 3 („the hub tells you”), built 2026-11-14 night, NOT deployed. Every new alarm is operator-only.**
|
||||
- **Two end dates warn 14 days before (R-1379, D77, `09` §3 decision 298; `08` §12).** `monitor.EndDateChecker`, hourly
|
||||
on the sweep: `operator_token_ends_soon` for an operator access token that is not revoked, and `web_cert_ends_soon`
|
||||
for a web certificate a box reports in `web_certs` (controllers after 0.352.1). Told once at 14 days and once more,
|
||||
as an error, if the date passes with the same thing still there; a renewed certificate has a new end date, so
|
||||
nothing clears. A box's names and a token's name are checked before they reach a mail; the box's list is cut at 8.
|
||||
A thing that ended more than 14 days ago raises nothing. Operator-only. No new table. Tests `TestEndDates_*`,
|
||||
`TestEndDateEventsAreOperatorOnly`, the wiring in `TestR1383_MainWiresTheSilentFailureChecks`; 12 mutations seen
|
||||
red (two did not run at first and were done again).
|
||||
- **The System page shows the kernel clean-up and the security-profile refresh (R-1383; `11` §5.11.1, §5.12).**
|
||||
Two new cells per box. „Kernel clean-up”: from the clean-up report's new `kernel_cleanup` object (kept in the stored
|
||||
report: `osupdates.Report.KernelCleanup`) — removed, kept, space gained, the names in the title; amber on
|
||||
|
||||
@@ -946,6 +946,27 @@ func main() {
|
||||
return ""
|
||||
},
|
||||
dataStore.OSAlarmRaised, dataStore.SetOSAlarmRaised, hubEvent, logger)
|
||||
// R-1379 (D77, decision 298): 14 days before an operator token or a box's web certificate ends. Hourly, on the sweep.
|
||||
endDateChecker := monitor.NewEndDateChecker(
|
||||
func() ([]monitor.EndingToken, error) {
|
||||
toks, err := dataStore.ListAPITokens()
|
||||
var out []monitor.EndingToken
|
||||
for _, t := range toks {
|
||||
if t.RevokedAt == nil {
|
||||
out = append(out, monitor.EndingToken{ID: t.ID, Name: t.Name, ExpiresAt: t.ExpiresAt})
|
||||
}
|
||||
}
|
||||
return out, err
|
||||
},
|
||||
func() ([]monitor.BoxCertReport, error) {
|
||||
cs, err := dataStore.GetCustomers()
|
||||
out := make([]monitor.BoxCertReport, 0, len(cs))
|
||||
for _, c := range cs {
|
||||
out = append(out, monitor.BoxCertReport{CustomerID: c.CustomerID, ReportJSON: c.ReportJSON})
|
||||
}
|
||||
return out, err
|
||||
},
|
||||
dataStore.OSAlarmRaised, dataStore.SetOSAlarmRaised, hubEvent, logger)
|
||||
// D53 (`08` §10): a held app reaches the OPERATOR — one reminder after the grace per reason, then weekly.
|
||||
var heldGrace time.Duration
|
||||
if v := cfg.Alerting.HeldAppGraceOverride; v != "" {
|
||||
@@ -989,6 +1010,7 @@ func main() {
|
||||
offsiteDeliveryChecker.Check()
|
||||
controllerSupervisorChecker.Check()
|
||||
signedJobWaitChecker.Check() // R-1383: a signed job a box did not take in two hours
|
||||
endDateChecker.Check() // R-1379: once an hour inside
|
||||
heldAppsChecker.Check() // D53: throttled to one pass per 15 min
|
||||
if goldenVouchChecker != nil {
|
||||
goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min
|
||||
|
||||
@@ -60,12 +60,14 @@ func TestR1383_MainWiresTheSilentFailureChecks(t *testing.T) {
|
||||
seen["lost-mail scan"]++
|
||||
case recv == "signedJobWaitChecker" && name == "Check":
|
||||
seen["signed-job sweep"]++
|
||||
case recv == "endDateChecker" && name == "Check":
|
||||
seen["end-date sweep"]++ // R-1379
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
for _, k := range []string{"ledger built", `watched "deadline-check"`, `watched "offsite-key-audit"`, "lost-mail scan", "signed-job sweep"} {
|
||||
for _, k := range []string{"ledger built", `watched "deadline-check"`, `watched "offsite-key-audit"`, "lost-mail scan", "signed-job sweep", "end-date sweep"} {
|
||||
if seen[k] != 1 {
|
||||
t.Errorf("main.go: %s — found %d time(s), want 1", k, seen[k])
|
||||
}
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// EndDateChecker (R-1379, D77, `09` §3 decision 298; `08-alarm-ladder.md`) warns the operator 14 days before two end
|
||||
// dates the product itself can read. Until it, no part of the product read either, and the first sign was the
|
||||
// failure on the day:
|
||||
//
|
||||
// - an OPERATOR ACCESS TOKEN (hub §17): 90 days by default; on its end date the release step that raises a floor
|
||||
// answers 401. The hub knows the date (api_tokens.expires_at).
|
||||
// - a box's WEB CERTIFICATE: only the box knows the date (it is in Traefik's store in the guest; the household's
|
||||
// public names end at Cloudflare). The controller carries it in its report (`web_certs`, controllers after
|
||||
// 0.352.1). Traefik renews 30 days before the end by itself — so 14 days left means the renewal has failed for
|
||||
// two weeks (a Cloudflare token of the household that ended, a name check that fails).
|
||||
//
|
||||
// One event per (token) and per (household, name, end date): told once at 14 days, and once more — as an ERROR — if
|
||||
// the date passes with the same certificate or token still there. A renewed certificate has a new end date, so
|
||||
// nothing has to clear. State: the hub's time-by-key settings, no new table. Operator-only. Pinned by TestEndDates_*.
|
||||
type EndDateChecker struct {
|
||||
tokens func() ([]EndingToken, error)
|
||||
reports func() ([]BoxCertReport, error)
|
||||
told func(key string) time.Time
|
||||
setTold func(key string, t time.Time) error
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
now func() time.Time
|
||||
every time.Duration
|
||||
warnDays int
|
||||
|
||||
mu sync.Mutex
|
||||
lastRun time.Time
|
||||
}
|
||||
|
||||
// EndingToken is one operator access token that is still valid to be used (not revoked).
|
||||
type EndingToken struct {
|
||||
ID int64
|
||||
Name string
|
||||
ExpiresAt time.Time
|
||||
}
|
||||
|
||||
// BoxCertReport is one household's newest controller report (raw), from which `web_certs` is read.
|
||||
type BoxCertReport struct {
|
||||
CustomerID string
|
||||
ReportJSON string
|
||||
}
|
||||
|
||||
const (
|
||||
EventTokenEndsSoon = "operator_token_ends_soon"
|
||||
EventWebCertEndsSoon = "web_cert_ends_soon"
|
||||
endDateWarnDays = 14
|
||||
endDateCheckEvery = time.Hour
|
||||
webCertsReadMax = 8
|
||||
)
|
||||
|
||||
var endDateNameRe = regexp.MustCompile(`^[A-Za-z0-9*._-]{1,253}$`)
|
||||
|
||||
// a token's name is the operator's own text (hub Tokens page); anything else is shown as „token #id”.
|
||||
var endDateTokenNameRe = regexp.MustCompile(`^[A-Za-z0-9 ._-]{1,64}$`)
|
||||
|
||||
func NewEndDateChecker(tokens func() ([]EndingToken, error), reports func() ([]BoxCertReport, error),
|
||||
told func(string) time.Time, setTold func(string, time.Time) error, onEvent EventNotifyFunc, logger *log.Logger) *EndDateChecker {
|
||||
return &EndDateChecker{tokens: tokens, reports: reports, told: told, setTold: setTold, onEvent: onEvent, logger: logger,
|
||||
now: time.Now, every: endDateCheckEvery, warnDays: endDateWarnDays}
|
||||
}
|
||||
|
||||
// Check runs on the hub's 60 s sweep and does its work once an hour.
|
||||
func (c *EndDateChecker) Check() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
now := c.now()
|
||||
if !c.lastRun.IsZero() && now.Sub(c.lastRun) < c.every {
|
||||
return
|
||||
}
|
||||
c.lastRun = now
|
||||
warn := time.Duration(c.warnDays) * 24 * time.Hour
|
||||
|
||||
// stage: "soon" inside the 14 days, "ended" from the end date on. Each stage is told once per key.
|
||||
tell := func(key string, end time.Time) (stage string, ok bool) {
|
||||
left := end.Sub(now)
|
||||
switch {
|
||||
case left > warn || left < -warn:
|
||||
return "", false // not yet — or ended so long ago that it is no news (an old token nobody replaced)
|
||||
case left > 0:
|
||||
stage = "soon"
|
||||
default:
|
||||
stage = "ended"
|
||||
}
|
||||
k := key + ":" + stage
|
||||
if !c.told(k).IsZero() {
|
||||
return stage, false
|
||||
}
|
||||
_ = c.setTold(k, now)
|
||||
return stage, true
|
||||
}
|
||||
daysLeft := func(end time.Time) int { return int(end.Sub(now) / (24 * time.Hour)) }
|
||||
|
||||
if toks, err := c.tokens(); err != nil {
|
||||
c.logger.Printf("[WARN] end-date check: cannot list the operator tokens: %v", err)
|
||||
} else {
|
||||
for _, t := range toks {
|
||||
stage, ok := tell(fmt.Sprintf("end_date:token:%d", t.ID), t.ExpiresAt)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
name := t.Name
|
||||
if !endDateTokenNameRe.MatchString(name) {
|
||||
name = fmt.Sprintf("token #%d", t.ID)
|
||||
}
|
||||
sev, msg := "warning", fmt.Sprintf("The operator access token „%s” ends in %d day(s), on %s. On that day every script that uses it gets 401 (for one: the release step that raises the floor). Do: make a new token on the hub's Tokens page and put it in the store of the program that uses it.",
|
||||
name, daysLeft(t.ExpiresAt), t.ExpiresAt.UTC().Format("2006-01-02"))
|
||||
if stage == "ended" {
|
||||
sev, msg = "error", fmt.Sprintf("The operator access token „%s” ENDED on %s and is not replaced: every script that uses it gets 401. Do: make a new token on the hub's Tokens page.",
|
||||
name, t.ExpiresAt.UTC().Format("2006-01-02"))
|
||||
}
|
||||
c.logger.Printf("[WARN] end-date check: operator token %d (%s) %s — ends %s", t.ID, name, stage, t.ExpiresAt.UTC().Format(time.RFC3339))
|
||||
details, _ := json.Marshal(map[string]any{"token_id": t.ID, "name": name, "expires_at": t.ExpiresAt.UTC().Format(time.RFC3339), "stage": stage})
|
||||
c.onEvent("", EventTokenEndsSoon, sev, msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
reps, err := c.reports()
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] end-date check: cannot read the reports: %v", err)
|
||||
return
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, r := range reps {
|
||||
if seen[r.CustomerID] {
|
||||
continue
|
||||
}
|
||||
seen[r.CustomerID] = true
|
||||
for _, wc := range parseWebCerts(r.ReportJSON) {
|
||||
stage, ok := tell("end_date:web_cert:"+r.CustomerID+":"+wc.Domain+":"+wc.NotAfter.UTC().Format("20060102T150405Z"), wc.NotAfter)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
sev, msg := "warning", fmt.Sprintf("The web certificate of %s for %s ends in %d day(s), on %s. The box renews it by itself 30 days before the end, so the renewal has FAILED for about two weeks. On the end date every app address of the household shows a browser warning. Do: read the Traefik log on the box for „acme”; the usual cause is the household's Cloudflare token.",
|
||||
r.CustomerID, wc.Domain, daysLeft(wc.NotAfter), wc.NotAfter.UTC().Format("2006-01-02"))
|
||||
if stage == "ended" {
|
||||
sev, msg = "error", fmt.Sprintf("The web certificate of %s for %s ENDED on %s and the box still reports it: every app address of the household shows a browser warning. Do: read the Traefik log on the box for „acme”.",
|
||||
r.CustomerID, wc.Domain, wc.NotAfter.UTC().Format("2006-01-02"))
|
||||
}
|
||||
c.logger.Printf("[WARN] end-date check: web certificate of %s for %s %s — ends %s", r.CustomerID, wc.Domain, stage, wc.NotAfter.UTC().Format(time.RFC3339))
|
||||
details, _ := json.Marshal(map[string]any{"domain": wc.Domain, "not_after": wc.NotAfter.UTC().Format(time.RFC3339), "stage": stage})
|
||||
c.onEvent(r.CustomerID, EventWebCertEndsSoon, sev, msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type webCert struct {
|
||||
Domain string
|
||||
NotAfter time.Time
|
||||
}
|
||||
|
||||
// parseWebCerts reads `web_certs` from a controller report: at most webCertsReadMax entries, each with a plain name
|
||||
// and a date. The list is the box's: a name that is not a host name, or a date that is not one, is dropped — the
|
||||
// box's words never reach a mail unchecked. An older controller sends none.
|
||||
func parseWebCerts(reportJSON string) []webCert {
|
||||
var w struct {
|
||||
WebCerts []struct {
|
||||
Domain string `json:"domain"`
|
||||
NotAfter string `json:"not_after"`
|
||||
} `json:"web_certs"`
|
||||
}
|
||||
if json.Unmarshal([]byte(reportJSON), &w) != nil {
|
||||
return nil
|
||||
}
|
||||
var out []webCert
|
||||
for _, c := range w.WebCerts {
|
||||
if len(out) == webCertsReadMax {
|
||||
break
|
||||
}
|
||||
at, err := time.Parse(time.RFC3339, c.NotAfter)
|
||||
if err != nil || !endDateNameRe.MatchString(c.Domain) || strings.Contains(c.Domain, "..") {
|
||||
continue
|
||||
}
|
||||
out = append(out, webCert{Domain: c.Domain, NotAfter: at})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].NotAfter.Before(out[j].NotAfter) })
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type endDateFix struct {
|
||||
now time.Time
|
||||
tokens []EndingToken
|
||||
reps []BoxCertReport
|
||||
state map[string]time.Time
|
||||
events []string
|
||||
c *EndDateChecker
|
||||
}
|
||||
|
||||
func newEndDateFix() *endDateFix {
|
||||
f := &endDateFix{now: time.Date(2026, 11, 14, 12, 0, 0, 0, time.UTC), state: map[string]time.Time{}}
|
||||
f.c = NewEndDateChecker(
|
||||
func() ([]EndingToken, error) { return f.tokens, nil },
|
||||
func() ([]BoxCertReport, error) { return f.reps, nil },
|
||||
func(k string) time.Time { return f.state[k] },
|
||||
func(k string, t time.Time) error { f.state[k] = t; return nil },
|
||||
func(cust, typ, sev, msg, details, src string) {
|
||||
f.events = append(f.events, cust+"|"+typ+"|"+sev+"|"+msg+"|"+details)
|
||||
}, log.New(io.Discard, "", 0))
|
||||
f.c.now = func() time.Time { return f.now }
|
||||
return f
|
||||
}
|
||||
|
||||
// step moves the clock (past the hourly throttle) and runs the check.
|
||||
func (f *endDateFix) step(d time.Duration) { f.now = f.now.Add(d); f.c.Check() }
|
||||
|
||||
func certReport(certs ...string) string { return `{"web_certs":[` + strings.Join(certs, ",") + `]}` }
|
||||
func cert(domain string, at time.Time) string {
|
||||
return fmt.Sprintf(`{"domain":%q,"not_after":%q,"issuer":"YR2"}`, domain, at.UTC().Format(time.RFC3339))
|
||||
}
|
||||
|
||||
// R-1379 (D77): the operator is warned 14 days before an end date — not at 15, once at 14, not again every hour, once
|
||||
// more (as an error) when the date passes with the same thing still there, and never for a token or certificate that
|
||||
// was renewed. The stored effect asserted is the event. Both kinds, one test each way.
|
||||
// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-end-dates.txt).
|
||||
func TestEndDates_FourteenDaysOnceThenEndedOnce(t *testing.T) {
|
||||
f := newEndDateFix()
|
||||
day := 24 * time.Hour
|
||||
tokEnd := f.now.Add(16*day + time.Hour)
|
||||
certEnd := f.now.Add(16*day + 2*time.Hour)
|
||||
f.tokens = []EndingToken{{ID: 7, Name: "CC hub token", ExpiresAt: tokEnd}, {ID: 8, Name: "fresh", ExpiresAt: f.now.Add(80 * day)}}
|
||||
f.reps = []BoxCertReport{{CustomerID: "demo-hp", ReportJSON: certReport(cert("*.enkisfelhom.hu", certEnd))},
|
||||
{CustomerID: "old-box", ReportJSON: `{"health":{"status":"ok"}}`}} // a controller that sends no web_certs
|
||||
f.step(0)
|
||||
f.step(day) // 15 days and a bit left
|
||||
if len(f.events) != 0 {
|
||||
t.Fatalf("at more than 14 days: %v", f.events)
|
||||
}
|
||||
f.step(2 * day) // 13 days and a bit left
|
||||
for i := 0; i < 30; i++ {
|
||||
f.step(2 * time.Hour)
|
||||
}
|
||||
if len(f.events) != 2 {
|
||||
t.Fatalf("inside the 14 days, 31 hourly checks: %d event(s), want ONE per end date: %v", len(f.events), f.events)
|
||||
}
|
||||
all := strings.Join(f.events, "\n")
|
||||
for _, want := range []string{"|operator_token_ends_soon|warning|", "„CC hub token”", tokEnd.Format("2006-01-02"), "401",
|
||||
"demo-hp|web_cert_ends_soon|warning|", "*.enkisfelhom.hu", certEnd.Format("2006-01-02"), "renewal has FAILED"} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("the events lack %q:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
if !strings.HasPrefix(f.events[0], "|") {
|
||||
t.Errorf("the token's event must carry no household: %s", f.events[0])
|
||||
}
|
||||
// the end dates pass, nothing was renewed: one ERROR each, once
|
||||
f.now = certEnd.Add(time.Hour)
|
||||
f.c.Check()
|
||||
f.step(2 * time.Hour)
|
||||
if len(f.events) != 4 || !strings.Contains(f.events[2]+f.events[3], "ENDED on") || strings.Count(f.events[2]+f.events[3], "|error|") != 2 {
|
||||
t.Fatalf("after the end date: %v", f.events[2:])
|
||||
}
|
||||
}
|
||||
|
||||
// A renewed certificate has a new end date: nothing is told. A certificate that is renewed AFTER the warning needs no
|
||||
// „all clear” — and a later failure of the new one is told again. Old things are no news: a token that ended months
|
||||
// ago, and nobody replaced, raises nothing when the checker first runs.
|
||||
func TestEndDates_RenewedAndLongGoneAreQuiet(t *testing.T) {
|
||||
f := newEndDateFix()
|
||||
day := 24 * time.Hour
|
||||
f.tokens = []EndingToken{{ID: 1, Name: "old", ExpiresAt: f.now.Add(-120 * day)}}
|
||||
f.reps = []BoxCertReport{{CustomerID: "n100", ReportJSON: certReport(cert("*.example.hu", f.now.Add(40*day)))}}
|
||||
f.step(0)
|
||||
f.step(2 * time.Hour)
|
||||
if len(f.events) != 0 {
|
||||
t.Fatalf("a token that ended 120 days ago and a certificate 40 days out: %v", f.events)
|
||||
}
|
||||
f.reps[0].ReportJSON = certReport(cert("*.example.hu", f.now.Add(10*day)))
|
||||
f.step(2 * time.Hour)
|
||||
if len(f.events) != 1 {
|
||||
t.Fatalf("10 days left: %v", f.events)
|
||||
}
|
||||
f.reps[0].ReportJSON = certReport(cert("*.example.hu", f.now.Add(89*day))) // renewed
|
||||
f.step(2 * time.Hour)
|
||||
f.step(11 * day)
|
||||
if len(f.events) != 1 {
|
||||
t.Fatalf("after the renewal: %v", f.events[1:])
|
||||
}
|
||||
f.step(70 * day) // the NEW certificate is now 8 days from its end
|
||||
if len(f.events) != 2 {
|
||||
t.Fatalf("the new certificate's own warning: %d event(s)", len(f.events))
|
||||
}
|
||||
}
|
||||
|
||||
// The list is the box's. A name that is not a host name and a date that is not a date are dropped, more than eight
|
||||
// entries are cut, and a token whose name is not plain text is named by its number.
|
||||
func TestEndDates_TheBoxesWordsAreChecked(t *testing.T) {
|
||||
f := newEndDateFix()
|
||||
soon := f.now.Add(5 * 24 * time.Hour)
|
||||
var many []string
|
||||
for i := 0; i < 20; i++ {
|
||||
many = append(many, cert(fmt.Sprintf("n%02d.example.hu", i), soon.Add(time.Duration(i)*time.Minute)))
|
||||
}
|
||||
f.reps = []BoxCertReport{
|
||||
{CustomerID: "evil", ReportJSON: certReport(cert(`<script>alert(1)</script>`, soon), cert("a b.example.hu", soon),
|
||||
`{"domain":"ok.example.hu","not_after":"tomorrow"}`, cert("fine.example.hu", soon))},
|
||||
{CustomerID: "many", ReportJSON: certReport(many...)},
|
||||
{CustomerID: "evil", ReportJSON: certReport(cert("second-report.example.hu", soon))}, // a duplicate row of the same household
|
||||
}
|
||||
f.tokens = []EndingToken{{ID: 9, Name: "<b>x</b>", ExpiresAt: soon}}
|
||||
f.step(0)
|
||||
all := strings.Join(f.events, "\n")
|
||||
if strings.Contains(all, "alert(1)") || strings.Contains(all, "a b.example") || strings.Contains(all, "<b>") || strings.Contains(all, "second-report") {
|
||||
t.Fatalf("an unchecked word of a box or a token name reached an event:\n%s", all)
|
||||
}
|
||||
if n := strings.Count(all, "evil|web_cert_ends_soon"); n != 1 || !strings.Contains(all, "fine.example.hu") {
|
||||
t.Errorf("household „evil”: %d event(s), want one for fine.example.hu", n)
|
||||
}
|
||||
if n := strings.Count(all, "many|web_cert_ends_soon"); n != webCertsReadMax {
|
||||
t.Errorf("household „many”: %d event(s), want the cap %d", n, webCertsReadMax)
|
||||
}
|
||||
if !strings.Contains(all, "„token #9”") {
|
||||
t.Errorf("a token with a name that is not plain text must be named by its number:\n%s", all)
|
||||
}
|
||||
}
|
||||
@@ -10,3 +10,12 @@ func TestBoxClockEventsAreOperatorOnly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-1379 (D77): the two end-date warnings are the operator's.
|
||||
func TestEndDateEventsAreOperatorOnly(t *testing.T) {
|
||||
for _, e := range []string{"operator_token_ends_soon", "web_cert_ends_soon"} {
|
||||
if !operatorOnlyEvents[e] {
|
||||
t.Errorf("%s must be operator-only", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -860,9 +860,12 @@ var operatorOnlyEvents = map[string]bool{
|
||||
"operator_mail_lost": true,
|
||||
"guest_power_gave_up": true,
|
||||
"guest_power_ok": true,
|
||||
"box_clock_wrong": true, // R-1384: a box's clock against the hub's — the operator's to read
|
||||
"box_clock_ok": true,
|
||||
"proxmox_trust_ok": true,
|
||||
// R-1379 (D77): an end date 14 days away — the operator's to act on.
|
||||
"operator_token_ends_soon": true,
|
||||
"web_cert_ends_soon": true,
|
||||
"box_clock_wrong": true, // R-1384: a box's clock against the hub's — the operator's to read
|
||||
"box_clock_ok": true,
|
||||
"proxmox_trust_ok": true,
|
||||
// R-836 (hub v0.143.0): the kernel lane's step results and the day-before notice's own record — operator facts. The
|
||||
// household's word is the day-before mail itself (SendKernelNotice), never these events.
|
||||
"os_kernel_step": true,
|
||||
|
||||
Reference in new issue
Block a user