diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index a862598c..30f072bd 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -1003,6 +1003,18 @@ a box that is fully blind sends no host report at all, so that case still reache LAST, in the same write that sets `ca_file`; the installer writes a fingerprint only when the copy was not proven; and a config that holds both is a pinned box — the agent treats it as „not moved" and runs the move. +**6. What an independent review changed (2026-11-14 night, built, not released).** (a) **The copy holds one or two +certificates.** The root step writes the new root FIRST and keeps the copy's previous first certificate behind it: +the copy is replaced before anything can be proven with the new root, and Proxmox serves its old certificate until +pveproxy is reloaded — without the old root, an agent that restarted in that window had no root that verified the +served certificate and no fingerprint. Never more than two; an ended one verifies nothing. (b) **One run at a time** +(`Ensure` is called from the start, the timer, a failed collect and a bundle install), and the root step has a +90-second deadline — it runs on the daemon's start path and inside the report loop. (c) **One lock for every writer +of `agent.json`** in the process (`config.FileMu`: the trust move, the PBS seed, the backup-target move): each reads +the whole file and writes the whole file. **Left open, low:** the write is in place (not a rename), and nothing reads +the parked copy `agent.json.pre-pve-trust` back by itself — on a full disk a failed write and a failed put-back can +leave an empty config; the hand repair is to copy the parked file back. + **What changes for a pinned box when Proxmox makes a new certificate:** the first failed call names `pinned_cert_changed`, runs the order above, and the next call works — the box is blind for one report cycle at most, not until a person repairs it. `runbooks/RUNBOOK-agent-proxmox-pin-repair.md` stays the hand repair for an agent older diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 0e31fc25..4e00da2e 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -432,6 +432,10 @@ label (`audits/RETIRE-peti-2026-09-25.md` A3). `runbooks/household-leaves.md`, its descriptor (`enabled:false`, coordinates kept — `TestPBSDR_DisableKeepsTenancy`), and RESET and delete take it down by the customer id, ticked or not. +**After review (2026-11-14):** a tick with the OTHER type (shared, then dedicated) makes a second account, so the +first account's record stays, and a RESET takes down every account type the config knows; the record is dropped only +after a provider leg that really ran — a leg skipped because this hub has no provider token leaves the record. + **Not proven:** on a deployed hub, or against the real provider. The tests run the real save, the real RESET and the repository's fake provider (`hetznerapi.Fake`); 9 mutations seen red, one of them only after a missing test case was added (`audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt`). @@ -724,6 +728,11 @@ Four cases, the four the ruling names. Each leaves an event row and mails the op queue, so a box cannot make operator mails by clearing job ids it invents; the give-up list is read only for guests the same report lists (not running), 16 entries at most. +**The mails (after review, 2026-11-14):** each of these events tells one thing once and keeps a mark, so its mail +cooldown is keyed per thing (`notify.perSubjectCooldownEvents`) — with the plain per-type key the second job, guest or +skipped check inside the hour was suppressed and never told. The lost-mail scan tells only mails that failed before +this hub process started (one that failed since still has its retry). One box holds at most 8 give-up marks. + **Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red). diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index 5993178d..32c3e73d 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -791,7 +791,10 @@ the product warns 14 days before the two it can read itself; the rest stays a li What it does not do: a box with NO certificate in the store (no ACME e-mail: Traefik serves its own certificate) is not told — the field is absent. A box that is switched off is judged on its last report: the warning is still true (an off box cannot renew). A token or certificate that ended more than 14 days ago raises nothing at the first run (no news). -**Not proven:** on a box or a deployed hub; a real failed renewal has never been seen. +After review (2026-11-14): the FIRST run after a deploy marks what has already ended and does not mail it; the box +reports only certificates of the household's current domain; a name is told again only for an end date at least a +day later than the one told, and one household holds at most 24 marks — a box cannot make mails or marks by sending +another date or new names each hour. **Not proven:** on a box or a deployed hub; a real failed renewal has never been seen. ### 12.2 The other sixteen `[DESIGN — proposed, not built]` diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 793db264..1b0d4d38 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -734,7 +734,10 @@ so the box's last refresh is still told after an agent restart), and `last_ask` this agent process ended (refreshed · not_needed · failed · refused · no_answer). The agent reads one small file and does not call the wrapper for it; a name that is not a plain unit name is dropped. The System page's cell **„Security profiles”** reads „7.0.14-28-pve · 2 rewritten · 18 h ago · restarted chrony.service”; amber, and under -„Needs attention”, when the last ask failed or got no answer. A refused ask (an older wrapper, a BYO host) is not a +„Needs attention”, when the last ask failed or got no answer. **After review (2026-11-14):** the wrapper's record lies in a +folder that is root-only on a box where a signed job made the folder first (read in source; to be read on a box), so +the agent takes the last refresh from the wrapper's ANSWER to its own ask — the wrapper answers its record at every +ask, also „not needed” — and the file only adds the detail where it can be read; `rewritten` is absent when not known. A refused ask (an older wrapper, a BYO host) is not a fault. No alarm is raised for a restart: it is shown, not mailed. Not seen on a box or a deployed hub. **Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced diff --git a/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt b/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt new file mode 100644 index 00000000..d71c0b32 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt @@ -0,0 +1,69 @@ +== MUTATION: the trust move writes agent.json without the shared lock + *** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0) + FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent [build failed] + +1 mutations, 1 not a red-proof +== MUTATION: a give-up stays reported after onboot is switched off + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s) + guestpower_test.go:315: onboot switched off: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}] +== MUTATION: a give-up stays reported for a locked guest + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s) + guestpower_test.go:315: locked by another operation: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}] +== MUTATION: a give-up stays reported for a destroyed guest + *** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0) + FAIL gitea.dooplex.hu/admin/felhom-agent/internal/localapi [build failed] + internal/localapi/guestpower.go:162:6: declared and not used: vmid + +3 mutations, 1 not a red-proof +== MUTATION: the two tokens come from two reads of the init + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestBootKey_AClockStepDoesNotMoveIt (0.00s) + guest_boot_key_test.go:84: GuestBootTokens = "1794112550-4242", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" after 2 read(s) of the init — want both tokens from ONE read + +1 mutations, 0 not a red-proof +== MUTATION: the stanza knows the last refresh only from the record file (root-only on a box) + RED on the mutant, GREEN on the real file + --- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s) + profilerefresh_test.go:139: stanza after an agent restart (the wrapper's answer, no detail) = {"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"} +== MUTATION: a refresh by this process is not remembered + RED on the mutant, GREEN on the real file + --- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s) + profilerefresh_test.go:123: stanza after a refresh, record file unreadable = {"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"} +== MUTATION: a readable record does not add the detail back + RED on the mutant, GREEN on the real file + --- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s) + profilerefresh_test.go:150: stanza with a readable record = {"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"} + +3 mutations, 0 not a red-proof +== MUTATION: the trust move writes agent.json without the shared lock — done again (the first mutant left an import unused) + RED on the mutant, GREEN on the real file + --- FAIL: TestAgentJSONWritersTakeTheSharedLock (0.00s) + config_writers_lock_test.go:55: ../../internal/pvetrust/pvetrust.go writeConfig: must start with config.FileMu.Lock() and defer config.FileMu.Unlock() + +1 mutations, 0 not a red-proof +== MUTATION: a give-up stays reported for a destroyed guest — done again (the first mutant left a variable unused) + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s) + guestpower_test.go:315: destroyed: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}] + +1 mutations, 0 not a red-proof +== MUTATION: the copy's previous certificate is thrown away at a change (the old behaviour: no root verifies the served certificate after a restart) + RED on the mutant: 3 failing line(s) + FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy) + AssertionError: Lists differ: [b'--[86 chars]----'] != [b'--[86 chars]----', b'-----BEGIN CERTIFICATE-----\nQUJDREVG[52 chars]---'] + FAILED (failures=1) +== MUTATION: the same source is written again at every run + RED on the mutant: 5 failing line(s) + FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy) + AssertionError: Tuples differ: (0, 'written', [('/etc/felhom-agent/pve-root-c[108 chars]20)]) != (0, 'same', []) + FAIL: test_the_same_copy_is_not_written_again (__main__.PVECACopy.test_the_same_copy_is_not_written_again) +== MUTATION: the copy grows by one certificate at every change + RED on the mutant: 3 failing line(s) + FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy) + AssertionError: Lists differ: [b'--[183 chars]----', b'-----BEGIN CERTIFICATE-----\nQUJDREVG[52 chars]---'] != [b'--[183 chars]----'] + FAILED (failures=1) + +the real file: 0 failing line(s) +3 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/review/redproof-controller-review-fixes.txt b/documentation/audits/night-build-2026-11-14/review/redproof-controller-review-fixes.txt new file mode 100644 index 00000000..4e88df21 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/review/redproof-controller-review-fixes.txt @@ -0,0 +1,10 @@ +== MUTATION: every certificate in the store is reported, also for a name the box no longer serves + RED on the mutant, GREEN on the real file + --- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s) + web_certs_report_test.go:62: a certificate for ANOTHER domain was reported: [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu}] +== MUTATION: the domain is matched as a plain suffix + RED on the mutant, GREEN on the real file + --- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s) + web_certs_report_test.go:65: a domain that only ENDS the same way matched: [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu}] + +2 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt b/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt new file mode 100644 index 00000000..d3887a6c --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt @@ -0,0 +1,104 @@ +== MUTATION: the cooldown key has no subject (the second token of the day is never mailed) + RED on the mutant, GREEN on the real file + --- FAIL: TestSubjectCooldown_TwoThingsTwoMails (0.04s) + subject_cooldown_test.go:24: mails = 1, want two (one per token) and none for the repeat: ["Customer: \nEvent: operator_token_ends_soon\nSeverity: warning\nTime: 2026-11-14 19:02 CET\nMessage: token A ends\nDetails: {\"su +== MUTATION: every event type gets the subject suffix (other types' keys change) + RED on the mutant, GREEN on the real file + --- FAIL: TestSubjectCooldown_TwoThingsTwoMails (0.04s) + subject_cooldown_test.go:37: backup_failed: its cooldown key changed (":x") — only the listed types may + subject_cooldown_test.go:37: app_start_failed: its cooldown key changed (":x") — only the listed types may + subject_cooldown_test.go:37: tunnel_down: its cooldown key changed (":x") — only the listed types may + +2 mutations, 0 not a red-proof +== MUTATION: the box's mode word reaches the event unchecked + *** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0) + FAIL gitea.dooplex.hu/admin/felhom-hub/internal/api [build failed] +== MUTATION: the box's date text reaches the event unchecked + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_OnlyClosedWordsReachTheEvent (0.08s) + review_fixes_test.go:25: the box's own text "send money" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: other). The root CA copy ends in 3 day(s) (send money to X). Design: `03` §12a.{"ca_not_afte + review_fixes_test.go:29: unknown words must read as other / empty: Trust in Proxmox on h1: a cause this hub does not know (mode: other). The root CA copy ends in 3 day(s) (send money to X). Design: `03` §12a.{"ca_not_after":"send +== MUTATION: no cap on a box's give-up marks + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.05s) + review_fixes_test.go:63: round 0: 16 give-up marks for one box, want at most 8 +== MUTATION: a mark for a guest the box no longer lists stays for ever + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.04s) + review_fixes_test.go:72: marks for guests the box no longer lists: 8 (rows 8), want 0 + +4 mutations, 1 not a red-proof +== MUTATION: a cleared alarm leaves an empty row + *** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0) +== MUTATION: the key list matches with LIKE-style prefix of another id (no exact prefix) + RED on the mutant, GREEN on the real file + --- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s) + d73_gone_rows_test.go:168: web-certificate marks of cust-a: 2, want 0 + d73_gone_rows_test.go:168: web-certificate marks of cust-b: 2, want 1 + +2 mutations, 1 not a red-proof +== MUTATION: a household's web-certificate marks outlive it + RED on the mutant, GREEN on the real file + --- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s) + d73_gone_rows_test.go:168: web-certificate marks of cust-a: 1, want 0 +== MUTATION: the purge also removes the marks of a household whose id starts the same way + RED on the mutant, GREEN on the real file + --- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s) + d73_gone_rows_test.go:168: web-certificate marks of cust-a2: 0, want 1 + +2 mutations, 0 not a red-proof +== MUTATION: a first run mails ENDED for a token that ended before the deploy + RED on the mutant, GREEN on the real file + --- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s) + end_dates_test.go:173: the first run: [|operator_token_ends_soon|error|The operator access token „replaced” ENDED on 2026-11-11 and is not revoked: every script that still uses it gets 401. Do: if it was replaced, revoke it on the +== MUTATION: a box gets a new mail for every new end date + RED on the mutant, GREEN on the real file + --- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s) + end_dates_test.go:195: a box that moves its end date by a minute each hour got 48 mails in two days (want at most a few: a new mail needs a date a day later) +== MUTATION: no cap on a household's web-certificate marks + RED on the mutant, GREEN on the real file + --- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s) + end_dates_test.go:208: 192 invented names made 192 marks, want at most 24 for one household + +3 mutations, 0 not a red-proof +== MUTATION: a tick with the other type drops the first account's record + RED on the mutant, GREEN on the real file + --- FAIL: TestKeepAccountRecord_FourCases (0.00s) + account_record_test.go:47: unticked shared, then ticked as DEDICATED: the shared account's record stays: record , want user u1-sub3 unticked_at 2026-11-01T00:00:00Z ({"offsite":{"enabled":true,"type":"dedicated","user":"u9"," + account_record_test.go:47: switched from shared to dedicated in ONE save: the shared account gets a record: record , want user u1-sub3 unticked_at 2026-11-14T20:00:00Z ({"offsite":{"enabled":true,"type":"dedicated","user":"u9 + +1 mutations, 0 not a red-proof +== MUTATION: the record is dropped also when the provider leg was skipped + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo (0.05s) + d71_offsite_account_record_test.go:193: the provider leg was SKIPPED and the account's record is gone: {} +== MUTATION: the RESET takes down only the first account type + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo (0.05s) + d71_offsite_account_record_test.go:197: a household with a dedicated tier AND a kept shared record: the RESET would take down [dedicated], want both +== MUTATION: the record is never dropped after a real takedown + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.06s) + d71_offsite_account_record_test.go:131: the record outlived the account: {"logging":{"level":"debug"},"offsite_account":{"type":"shared","user":"u629193-sub1000","host":"u629193-sub1000.your-storagebox.de","unticked_at":"2026-11-1 + +3 mutations, 0 not a red-proof +== MUTATION: a mail that failed in this process is told as lost + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_AMailThatFailedInThisProcessIsNotLost (0.04s) + lost_operator_mail_test.go:109: a mail that failed AFTER this process started was told as lost (1) + +1 mutations, 0 not a red-proof +== MUTATION: a cleared alarm leaves an empty row — done again: the first run was GREEN (the test read through a function that hides empty rows; a test that counts the rows was added) + RED on the mutant, GREEN on the real file + --- FAIL: TestOSAlarm_AClearedMarkLeavesNoRow (0.04s) + d73_gone_rows_test.go:253: a cleared mark left 1 row(s) + +1 mutations, 0 not a red-proof +== MUTATION: the box's mode word reaches the event unchecked — done again (the first mutant did not compile) + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_OnlyClosedWordsReachTheEvent (0.08s) + review_fixes_test.go:25: the box's own text "MODE" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: MODE call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"" + review_fixes_test.go:25: the box's own text "" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: MODE call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"", + review_fixes_test.go:29: unknown words must read as other / empty: Trust in Proxmox on h1: a cause this hub does not know (mode: MODE call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"","c + +1 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/review/redproof-installer-review-fixes.txt b/documentation/audits/night-build-2026-11-14/review/redproof-installer-review-fixes.txt new file mode 100644 index 00000000..c2027262 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/review/redproof-installer-review-fixes.txt @@ -0,0 +1,9 @@ +== MUTATION: a failed fingerprint read ends the installer (the old line) + RED on the mutant: 1 failing line(s) + FAIL test_pve_trust_no_answer_from_proxmox_does_not_end_the_installer -- +== MUTATION: the proof is a pipe into grep -q again (a good copy can read as not verified under pipefail) + *** NOT A RED-PROOF ***: 0 failing line(s) + +the real file: 0 failing line(s) +2 mutations, 1 not a red-proof +NOTE: the second mutation is NOT a red-proof and is left here on purpose. The defect it models (grep -q closes the pipe while s_client still writes; pipefail then fails the pipeline for a good copy) was reasoned by the reviewer and could not be reproduced in this test in 2 runs. The fix (read the answer whole, search after) removes the pipe, so the question does not arise; it is NOT proven that the old line ever failed. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index e412681e..9899634a 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -3,6 +3,23 @@ - `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change. - **Release 3 („the hub tells you”), built 2026-11-14 night, NOT deployed. Every new alarm is operator-only.** +- **Fixes after an independent review of the night's code (2026-11-14; + `documentation/audits/night-build-2026-11-14/review/`).** (1) **Per-thing mail cooldown** for the told-once events + (`notify.perSubjectCooldownEvents`, details `subject`): the plain `customer:type` key suppressed the second token, + certificate, job, guest or cause inside the hour, and its mark was already set, so it was never mailed. (2) What a + BOX sends is bounded and checked: the trust event takes only a closed mode word and a real date; one box holds at + most 8 give-up marks and a mark for a guest it no longer lists is dropped; web-certificate marks are per (household, + name, stage), hold the end date told, are capped at 24 per household, and a name is told again only for an end + date a day later. (3) `SetOSAlarmRaised` with no time DELETES the row (a cleared mark left an empty row for ever). + (4) The first run of the end-date check marks what has already ended and does not mail it. (5) The off-site record: + a tick with the OTHER type keeps the first account's record, a RESET takes down every account type the config + knows, and the record is dropped only after a provider leg that really ran. (6) The lost-mail scan tells only mails + that failed before THIS process started, and cuts text by rune. (7) A deleted household's web-certificate marks go + with it. (8) The „job not taken” text names the fourth cause (an agent too old for that job). Tests + `TestSubjectCooldown_*`, `TestProxmoxTrust_OnlyClosedWordsReachTheEvent`, `TestGuestPowerGaveUp_ABoxCannotFillTheSettings`, + `TestOSAlarm_AClearedMarkLeavesNoRow`, `TestEndDates_FirstRunAndABoxCannotMintMarks`, + `TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo`, `TestLostOperatorMail_AMailThatFailedInThisProcessIsNotLost`; + 20 mutations seen red (two after a redo; one of those because the test could not see the defect and was extended). - **A deleted box or household leaves no rows in the tables that had no delete (R-1376 part 1, D73, `09` §3 decision 293; `05` §14.2).** `DeleteHost` also removes the box's `os_host_settings`, `os_reports`, `os_kernel_notices` and the hub's alarm marks that name it; `PurgeCustomerResidue` also removes the household's `offsite_keys`, diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index df40c267..38b475f4 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -928,7 +928,7 @@ func main() { // R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop. controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger) // R-1383 (2) an operator mail that was never sent — told at this start, in one mail (after the mail hold is wired). - go dispatcher.ReportLostOperatorMails(time.Now()) + go dispatcher.ReportLostOperatorMails(time.Now(), processStart) // R-1383 (3) a signed job a box never took — on the 60 s sweep. signedJobWaitChecker := monitor.NewSignedJobWaitChecker( func() ([]monitor.PendingSignedJob, error) { @@ -966,7 +966,7 @@ func main() { } return out, err }, - dataStore.OSAlarmRaised, dataStore.SetOSAlarmRaised, hubEvent, logger) + dataStore.OSAlarmRaised, dataStore.SetOSAlarmRaised, dataStore.OSAlarmKeys, hubEvent, logger) // D53 (`08` §10): a held app reaches the OPERATOR — one reminder after the grace per reason, then weekly. var heldGrace time.Duration if v := cfg.Alerting.HeldAppGraceOverride; v != "" { @@ -1133,6 +1133,10 @@ func loadConfig(path string, logger *log.Logger) *Config { return cfg } +// processStart is when this hub process began. The lost-mail scan tells only mails that failed BEFORE it: a mail +// that failed since still has this process's retry timers. +var processStart = time.Now() + // dailyLedger remembers across a restart when a watched daily check last ran (R-1383). nil until main wires it; a nil // ledger is safe to call. var dailyLedger *monitor.DailyLedger diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 17af3fb5..4814cb34 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -14,6 +14,7 @@ import ( "net/http" "regexp" "sort" + "strconv" "strings" "sync" "time" @@ -3309,6 +3310,9 @@ var signedJobIDRe = regexp.MustCompile(`^[A-Za-z0-9._-]{1,64}$`) // guestPowerMaxEntries bounds the give-up list read from ONE report (a box has a handful of guests). const guestPowerMaxEntries = 16 +// guestPowerMaxMarks bounds how many give-up marks ONE box can hold in the hub's settings at a time. +const guestPowerMaxMarks = 8 + func (h *Handler) noteSignedJobOutcome(hostID, jobID, outcome, reason, op string) { switch outcome { case "rejected", "failed", "malformed": @@ -3337,7 +3341,7 @@ func (h *Handler) noteSignedJobOutcome(hostID, jobID, outcome, reason, op string } h.logger.Printf("[WARN] host %s signed job %s (%s): %s — %s", hostID, jobID, op, outcome, reason) msg := fmt.Sprintf("The signed job %s (%s) for %s was %s: %s. The job is gone from the queue and will not run again. Do: read the cause, then sign a new job.", jobID, op, hostID, what, text) - details, _ := json.Marshal(map[string]any{"host_id": hostID, "job_id": jobID, "op": op, "outcome": outcome, "reason": reason}) + details, _ := json.Marshal(map[string]any{"host_id": hostID, "job_id": jobID, "op": op, "outcome": outcome, "reason": reason, "subject": hostID + ":" + jobID}) if _, err := h.store.SaveEvent(custID, eventSignedJobRefused, "error", msg, string(details), "hub"); err != nil { h.logger.Printf("[WARN] %s event save FAILED for %s: %v", eventSignedJobRefused, hostID, err) } else if h.dispatcher != nil { @@ -3373,25 +3377,47 @@ func (h *Handler) checkGuestPowerGaveUp(hostID, custID string, gave []hostGuestP go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(dj), "hub") } } - key := func(vmid int) string { return fmt.Sprintf("guest_power:%s:%d", hostID, vmid) } - for _, g := range gave { - if g.VMID <= 0 || !known[g.VMID] || running[g.VMID] || !h.store.OSAlarmRaised(key(g.VMID)).IsZero() { + prefix := fmt.Sprintf("guest_power:%s:", hostID) + key := func(vmid int) string { return prefix + strconv.Itoa(vmid) } + // The marks this box has now. The report's own guest list is the BOX's word too, so „the report lists the guest” + // bounds nothing by itself (review, 2026-11-14): the number of marks per box is capped, and a mark for a guest the + // box no longer lists is dropped — a destroyed guest must not hold a mark, or a slot, for ever. + raised := map[int]bool{} + for _, k := range h.store.OSAlarmKeys(prefix) { + vmid, err := strconv.Atoi(strings.TrimPrefix(k, prefix)) + if err != nil { continue } + if !known[vmid] { + _ = h.store.SetOSAlarmRaised(k, time.Time{}) + h.logger.Printf("[INFO] host %s: guest %d is no longer reported — its give-up mark is dropped", hostID, vmid) + continue + } + raised[vmid] = true + } + for _, g := range gave { + if g.VMID <= 0 || !known[g.VMID] || running[g.VMID] || raised[g.VMID] { + continue + } + if len(raised) >= guestPowerMaxMarks { + h.logger.Printf("[WARN] host %s: %d give-up marks already — guest %d is not told (a box has a handful of guests)", hostID, len(raised), g.VMID) + break + } + raised[g.VMID] = true _ = h.store.SetOSAlarmRaised(key(g.VMID), time.Now()) h.logger.Printf("[WARN] host %s: the guest-power watchdog gave up on guest %d (%d attempts)", hostID, g.VMID, g.Attempts) emit(eventGuestPowerGaveUp, "error", fmt.Sprintf("Guest %d on %s is STOPPED and the agent gave up starting it after %d failed attempts. The household's apps are off. The agent does not try again by itself. Do: read `journalctl -u felhom-agent` on the box for „guest-power”, start the guest, and read why it did not start.", g.VMID, hostID, g.Attempts), - map[string]any{"host_id": hostID, "vmid": g.VMID, "attempts": g.Attempts, "since": g.Since}) + map[string]any{"host_id": hostID, "vmid": g.VMID, "attempts": g.Attempts, "since": trustDate(g.Since), "subject": key(g.VMID)}) } for vmid := range running { - if h.store.OSAlarmRaised(key(vmid)).IsZero() { + if !raised[vmid] { continue } _ = h.store.SetOSAlarmRaised(key(vmid), time.Time{}) h.logger.Printf("[INFO] host %s: guest %d runs again (the guest-power watchdog had given up)", hostID, vmid) emit(eventGuestPowerRunning, "info", fmt.Sprintf("Guest %d on %s runs again.", vmid, hostID), - map[string]any{"host_id": hostID, "vmid": vmid}) + map[string]any{"host_id": hostID, "vmid": vmid, "subject": key(vmid)}) } } @@ -3542,12 +3568,13 @@ func (h *Handler) checkProxmoxTrust(hostID, custID string, t *hostProxmoxTrust) } if len(cleared) > 0 { sort.Strings(cleared) - h.logger.Printf("[INFO] host %s proxmox trust: no problem any more (was: %s; mode %s)", hostID, strings.Join(cleared, ", "), t.Mode) - emit(eventProxmoxTrustOK, "info", fmt.Sprintf("The agent of %s trusts Proxmox again (mode: %s). Was: %s.", hostID, t.Mode, strings.Join(cleared, ", ")), - map[string]any{"host_id": hostID, "mode": t.Mode, "was": cleared}) + h.logger.Printf("[INFO] host %s proxmox trust: no problem any more (was: %s; mode %s)", hostID, strings.Join(cleared, ", "), trustModeWord(t.Mode)) + emit(eventProxmoxTrustOK, "info", fmt.Sprintf("The agent of %s trusts Proxmox again (mode: %s). Was: %s.", hostID, trustModeWord(t.Mode), strings.Join(cleared, ", ")), + map[string]any{"host_id": hostID, "mode": trustModeWord(t.Mode), "was": cleared}) } return } + mode, caEnds := trustModeWord(t.Mode), trustDate(t.CANotAfter) cause, text := t.Problem, proxmoxTrustProblems[t.Problem] if text == "" { cause, text = "other", "a cause this hub does not know" @@ -3563,11 +3590,28 @@ func (h *Handler) checkProxmoxTrust(hostID, custID string, t *hostProxmoxTrust) } days := "" if t.CADaysLeft != nil { - days = fmt.Sprintf(" The root CA copy ends in %d day(s) (%s).", *t.CADaysLeft, t.CANotAfter) + days = fmt.Sprintf(" The root CA copy ends in %d day(s) (%s).", *t.CADaysLeft, caEnds) } - h.logger.Printf("[WARN] host %s proxmox trust: %s (mode %s)", hostID, cause, t.Mode) - emit(eventProxmoxTrustProblem, sev, fmt.Sprintf("Trust in Proxmox on %s: %s (mode: %s).%s Design: `03` §12a.", hostID, text, t.Mode, days), - map[string]any{"host_id": hostID, "cause": cause, "mode": t.Mode, "reported_cause": truncateRunes(t.Problem, 60), "ca_not_after": t.CANotAfter}) + h.logger.Printf("[WARN] host %s proxmox trust: %s (mode %s)", hostID, cause, mode) + emit(eventProxmoxTrustProblem, sev, fmt.Sprintf("Trust in Proxmox on %s: %s (mode: %s).%s Design: `03` §12a.", hostID, text, mode, days), + map[string]any{"host_id": hostID, "cause": cause, "mode": mode, "ca_not_after": caEnds, "subject": hostID + ":" + cause}) +} + +// trustModeWord / trustDate: the stanza is the BOX's. Only a word of the closed list and a real date reach an event +// (and so a mail); anything else is „other” / "" — the comment „never trusted as text” must hold for every field. +func trustModeWord(m string) string { + switch m { + case "fingerprint", "ca", "ca_file_own", "insecure", "system": + return m + } + return "other" +} + +func trustDate(s string) string { + if t, err := time.Parse(time.RFC3339, s); err == nil { + return t.UTC().Format(time.RFC3339) + } + return "" } func truncateRunes(s string, n int) string { diff --git a/hub/internal/api/review_fixes_test.go b/hub/internal/api/review_fixes_test.go new file mode 100644 index 00000000..256fd5da --- /dev/null +++ b/hub/internal/api/review_fixes_test.go @@ -0,0 +1,80 @@ +package api + +import ( + "net/http" + "strings" + "testing" + "time" +) + +// Review of 2026-11-14. The trust stanza is the BOX's: only a word of the closed list and a real date may reach the +// event (and its details, which the mail appends). The first version passed `mode` and `ca_not_after` through. +func TestProxmoxTrust_OnlyClosedWordsReachTheEvent(t *testing.T) { + h, st := trustSetup(t) + bad := `{"mode":"MODE call +36…","problem":"FREE TEXT problem","detail":"x","ca_days_left":3,"ca_not_after":"send money to X"}` + if rr := do(h, http.MethodPost, "/host-report", "HKEY", trustReport(bad)); rr.Code != 200 { + t.Fatalf("report: %d", rr.Code) + } + evs := r1383Events(t, st, eventProxmoxTrustProblem) + if len(evs) != 1 { + t.Fatalf("events = %d", len(evs)) + } + all := evs[0].Message + evs[0].DetailsJSON + for _, leak := range []string{"MODE", "", "send money", "FREE TEXT"} { + if strings.Contains(all, leak) { + t.Errorf("the box's own text %q reached the event: %s", leak, all) + } + } + if !strings.Contains(all, "mode: other") || !strings.Contains(all, `"cause":"other"`) || !strings.Contains(all, `"ca_not_after":""`) { + t.Errorf("unknown words must read as other / empty: %s", all) + } + // and a real stanza is unchanged + h2, st2 := trustSetup(t) + good := `{"mode":"fingerprint","problem":"ca_copy_does_not_verify","ca_days_left":3560,"ca_not_after":"2036-08-17T15:44:42Z"}` + do(h2, http.MethodPost, "/host-report", "HKEY", trustReport(good)) + e := r1383Events(t, st2, eventProxmoxTrustProblem)[0] + if !strings.Contains(e.Message, "mode: fingerprint") || !strings.Contains(e.Message, "2036-08-17T15:44:42Z") { + t.Errorf("a real stanza lost its words: %s", e.Message) + } +} + +// The report's own guest list is the box's word too, so „only a guest the report lists” bounds nothing by itself. One +// box holds at most guestPowerMaxMarks give-up marks; a mark for a guest the box no longer lists is dropped — and a +// dropped mark leaves NO row behind (a box could otherwise fill the settings table with empty rows). +func TestGuestPowerGaveUp_ABoxCannotFillTheSettings(t *testing.T) { + h, st := trustSetup(t) + marks := func() int { return len(st.OSAlarmKeys("guest_power:h1:")) } + rows := func() int { + var n int + for _, k := range []string{"guest_power:h1:"} { + n += len(st.OSAlarmKeys(k)) + } + return n + } + for round := 0; round < 5; round++ { // five reports, each with sixteen NEW invented guests, all "stopped" + known, gave := map[int]bool{}, []hostGuestPowerGaveUp{} + for i := 0; i < 16; i++ { + vmid := 7000 + round*100 + i + known[vmid] = true + gave = append(gave, hostGuestPowerGaveUp{VMID: vmid, Since: "2026-11-14T02:00:00Z", Attempts: 3}) + } + h.checkGuestPowerGaveUp("h1", "c1", gave, map[int]bool{}, known) + if m := marks(); m > guestPowerMaxMarks { + t.Fatalf("round %d: %d give-up marks for one box, want at most %d", round, m, guestPowerMaxMarks) + } + } + if n := len(r1383Events(t, st, eventGuestPowerGaveUp)); n > 5*guestPowerMaxMarks { + t.Fatalf("%d events for five reports", n) + } + // the box now reports only its real guest: every invented mark is gone, and so is its row + h.checkGuestPowerGaveUp("h1", "c1", nil, map[int]bool{100: true}, map[int]bool{100: true}) + if marks() != 0 || rows() != 0 { + t.Fatalf("marks for guests the box no longer lists: %d (rows %d), want 0", marks(), rows()) + } + // a cleared mark leaves no row at all + _ = st.SetOSAlarmRaised("probe:x", time.Now()) + _ = st.SetOSAlarmRaised("probe:x", time.Time{}) + if !st.OSAlarmRaised("probe:x").IsZero() || len(st.OSAlarmKeys("probe:")) != 0 { + t.Fatalf("a cleared alarm is still there") + } +} diff --git a/hub/internal/monitor/daily_ledger.go b/hub/internal/monitor/daily_ledger.go index b3c75fbd..392724ec 100644 --- a/hub/internal/monitor/daily_ledger.go +++ b/hub/internal/monitor/daily_ledger.go @@ -82,7 +82,7 @@ func (l *DailyLedger) Started(name string, now time.Time, hour, min int, loc *ti msg := fmt.Sprintf("The hub's daily check „%s” did NOT run at %s: the hub was not running at that minute. It last ran at %s (%d slot(s) missed). The hub does not run it late. %s", name, prev.Format("2006-01-02 15:04 MST"), last.In(loc).Format("2006-01-02 15:04 MST"), missed, dailySkipAdvice(name)) details, _ := json.Marshal(map[string]any{"job": name, "missed_slot": prev.UTC().Format(time.RFC3339), - "last_run": last.UTC().Format(time.RFC3339), "missed": missed}) + "last_run": last.UTC().Format(time.RFC3339), "missed": missed, "subject": name}) if l.onEvent != nil { l.onEvent("", EventDailyJobSkipped, "warning", msg, string(details), "hub") } diff --git a/hub/internal/monitor/end_dates.go b/hub/internal/monitor/end_dates.go index 91459861..97481034 100644 --- a/hub/internal/monitor/end_dates.go +++ b/hub/internal/monitor/end_dates.go @@ -22,14 +22,15 @@ import ( // 0.352.1). Traefik renews 30 days before the end by itself — so 14 days left means the renewal has failed for // two weeks (a Cloudflare token of the household that ended, a name check that fails). // -// One event per (token) and per (household, name, end date): told once at 14 days, and once more — as an ERROR — if -// the date passes with the same certificate or token still there. A renewed certificate has a new end date, so -// nothing has to clear. State: the hub's time-by-key settings, no new table. Operator-only. Pinned by TestEndDates_*. +// One event per token and per (household, name): told once at 14 days, and once more — as an ERROR — if the date +// passes with the same certificate or token still there. A renewed certificate has a later end date and is told +// again only when THAT one comes within 14 days, so nothing has to clear. State: the hub's time-by-key settings, no new table. Operator-only. Pinned by TestEndDates_*. type EndDateChecker struct { tokens func() ([]EndingToken, error) reports func() ([]BoxCertReport, error) told func(key string) time.Time setTold func(key string, t time.Time) error + keys func(prefix string) []string // the marks under a prefix (store.OSAlarmKeys) onEvent EventNotifyFunc logger *log.Logger now func() time.Time @@ -59,6 +60,7 @@ const ( endDateWarnDays = 14 endDateCheckEvery = time.Hour webCertsReadMax = 8 + webCertMaxMarks = 24 // per household: names × two stages ) var endDateNameRe = regexp.MustCompile(`^[A-Za-z0-9*._-]{1,253}$`) @@ -67,8 +69,8 @@ var endDateNameRe = regexp.MustCompile(`^[A-Za-z0-9*._-]{1,253}$`) var endDateTokenNameRe = regexp.MustCompile(`^[A-Za-z0-9 ._-]{1,64}$`) func NewEndDateChecker(tokens func() ([]EndingToken, error), reports func() ([]BoxCertReport, error), - told func(string) time.Time, setTold func(string, time.Time) error, onEvent EventNotifyFunc, logger *log.Logger) *EndDateChecker { - return &EndDateChecker{tokens: tokens, reports: reports, told: told, setTold: setTold, onEvent: onEvent, logger: logger, + told func(string) time.Time, setTold func(string, time.Time) error, keys func(string) []string, onEvent EventNotifyFunc, logger *log.Logger) *EndDateChecker { + return &EndDateChecker{tokens: tokens, reports: reports, told: told, setTold: setTold, keys: keys, onEvent: onEvent, logger: logger, now: time.Now, every: endDateCheckEvery, warnDays: endDateWarnDays} } @@ -82,24 +84,19 @@ func (c *EndDateChecker) Check() { } c.lastRun = now warn := time.Duration(c.warnDays) * 24 * time.Hour + // The FIRST run ever (a first deploy): a thing that has ALREADY ended is marked and not told — an old token that + // was replaced long ago and never revoked is no news. What ends within the 14 days is told as always. + baseline := c.told("end_date:baseline").IsZero() - // stage: "soon" inside the 14 days, "ended" from the end date on. Each stage is told once per key. - tell := func(key string, end time.Time) (stage string, ok bool) { + stageOf := func(end time.Time) string { left := end.Sub(now) switch { case left > warn || left < -warn: - return "", false // not yet — or ended so long ago that it is no news (an old token nobody replaced) + return "" // not yet — or ended so long ago that it is no news case left > 0: - stage = "soon" - default: - stage = "ended" + return "soon" } - k := key + ":" + stage - if !c.told(k).IsZero() { - return stage, false - } - _ = c.setTold(k, now) - return stage, true + return "ended" } daysLeft := func(end time.Time) int { return int(end.Sub(now) / (24 * time.Hour)) } @@ -107,8 +104,16 @@ func (c *EndDateChecker) Check() { c.logger.Printf("[WARN] end-date check: cannot list the operator tokens: %v", err) } else { for _, t := range toks { - stage, ok := tell(fmt.Sprintf("end_date:token:%d", t.ID), t.ExpiresAt) - if !ok { + stage := stageOf(t.ExpiresAt) + if stage == "" { + continue + } + k := fmt.Sprintf("end_date:token:%d:%s", t.ID, stage) + if !c.told(k).IsZero() { + continue + } + _ = c.setTold(k, now) + if baseline && stage == "ended" { continue } name := t.Name @@ -118,11 +123,12 @@ func (c *EndDateChecker) Check() { sev, msg := "warning", fmt.Sprintf("The operator access token „%s” ends in %d day(s), on %s. On that day every script that uses it gets 401 (for one: the release step that raises the floor). Do: make a new token on the hub's Tokens page and put it in the store of the program that uses it.", name, daysLeft(t.ExpiresAt), t.ExpiresAt.UTC().Format("2006-01-02")) if stage == "ended" { - sev, msg = "error", fmt.Sprintf("The operator access token „%s” ENDED on %s and is not replaced: every script that uses it gets 401. Do: make a new token on the hub's Tokens page.", + sev, msg = "error", fmt.Sprintf("The operator access token „%s” ENDED on %s and is not revoked: every script that still uses it gets 401. Do: if it was replaced, revoke it on the hub's Tokens page; if not, make a new token.", name, t.ExpiresAt.UTC().Format("2006-01-02")) } c.logger.Printf("[WARN] end-date check: operator token %d (%s) %s — ends %s", t.ID, name, stage, t.ExpiresAt.UTC().Format(time.RFC3339)) - details, _ := json.Marshal(map[string]any{"token_id": t.ID, "name": name, "expires_at": t.ExpiresAt.UTC().Format(time.RFC3339), "stage": stage}) + details, _ := json.Marshal(map[string]any{"token_id": t.ID, "name": name, "expires_at": t.ExpiresAt.UTC().Format(time.RFC3339), + "stage": stage, "subject": fmt.Sprintf("token:%d:%s", t.ID, stage)}) c.onEvent("", EventTokenEndsSoon, sev, msg, string(details), "hub") } } @@ -138,9 +144,31 @@ func (c *EndDateChecker) Check() { continue } seen[r.CustomerID] = true + // The names and dates are the BOX's. Its marks are keyed per (household, name, stage) and hold the END DATE + // that was told, so a box cannot mint a new mark, or a new mail, by sending another date each hour: a name is + // told again only for an end date at least a day LATER than the one told (a real renewal moves it by weeks), + // and one household holds at most webCertMaxMarks marks. (Review, 2026-11-14.) + prefix := "end_date:web_cert:" + r.CustomerID + ":" + marks := len(c.keys(prefix)) for _, wc := range parseWebCerts(r.ReportJSON) { - stage, ok := tell("end_date:web_cert:"+r.CustomerID+":"+wc.Domain+":"+wc.NotAfter.UTC().Format("20060102T150405Z"), wc.NotAfter) - if !ok { + stage := stageOf(wc.NotAfter) + if stage == "" { + continue + } + k := prefix + wc.Domain + ":" + stage + toldEnd := c.told(k) + if !toldEnd.IsZero() && !wc.NotAfter.After(toldEnd.Add(24*time.Hour)) { + continue + } + if toldEnd.IsZero() { + if marks >= webCertMaxMarks { + c.logger.Printf("[WARN] end-date check: %s already holds %d web-certificate marks — %s is not told", r.CustomerID, marks, wc.Domain) + continue + } + marks++ + } + _ = c.setTold(k, wc.NotAfter) + if baseline && stage == "ended" { continue } sev, msg := "warning", fmt.Sprintf("The web certificate of %s for %s ends in %d day(s), on %s. The box renews it by itself 30 days before the end, so the renewal has FAILED for about two weeks. On the end date every app address of the household shows a browser warning. Do: read the Traefik log on the box for „acme”; the usual cause is the household's Cloudflare token.", @@ -150,10 +178,14 @@ func (c *EndDateChecker) Check() { r.CustomerID, wc.Domain, wc.NotAfter.UTC().Format("2006-01-02")) } c.logger.Printf("[WARN] end-date check: web certificate of %s for %s %s — ends %s", r.CustomerID, wc.Domain, stage, wc.NotAfter.UTC().Format(time.RFC3339)) - details, _ := json.Marshal(map[string]any{"domain": wc.Domain, "not_after": wc.NotAfter.UTC().Format(time.RFC3339), "stage": stage, "issuer": wc.Issuer}) + details, _ := json.Marshal(map[string]any{"domain": wc.Domain, "not_after": wc.NotAfter.UTC().Format(time.RFC3339), "stage": stage, + "issuer": wc.Issuer, "subject": wc.Domain + ":" + stage}) c.onEvent(r.CustomerID, EventWebCertEndsSoon, sev, msg, string(details), "hub") } } + if baseline { + _ = c.setTold("end_date:baseline", now) + } } type webCert struct { diff --git a/hub/internal/monitor/end_dates_test.go b/hub/internal/monitor/end_dates_test.go index a2ce1348..a7b5a1f7 100644 --- a/hub/internal/monitor/end_dates_test.go +++ b/hub/internal/monitor/end_dates_test.go @@ -25,6 +25,15 @@ func newEndDateFix() *endDateFix { func() ([]BoxCertReport, error) { return f.reps, nil }, func(k string) time.Time { return f.state[k] }, func(k string, t time.Time) error { f.state[k] = t; return nil }, + func(prefix string) []string { + var out []string + for k := range f.state { + if strings.HasPrefix(k, prefix) { + out = append(out, k) + } + } + return out + }, func(cust, typ, sev, msg, details, src string) { f.events = append(f.events, cust+"|"+typ+"|"+sev+"|"+msg+"|"+details) }, log.New(io.Discard, "", 0)) @@ -149,3 +158,53 @@ func TestEndDates_TheBoxesWordsAreChecked(t *testing.T) { t.Errorf("a token with a name that is not plain text must be named by its number:\n%s", all) } } + +// Found by review, 2026-11-14. (1) The FIRST run after a deploy: a token that already ended, was replaced long ago and +// never revoked, must not mail „ENDED” — it is marked, not told; a thing that ends within the 14 days is still told. +// (2) The marks are the hub's settings and the names and dates are the box's: a box that sends another end date every +// hour, or new names every hour, must not mint a mail or a mark each time. +func TestEndDates_FirstRunAndABoxCannotMintMarks(t *testing.T) { + day := 24 * time.Hour + f := newEndDateFix() + f.tokens = []EndingToken{{ID: 1, Name: "replaced", ExpiresAt: f.now.Add(-3 * day)}, {ID: 2, Name: "soon", ExpiresAt: f.now.Add(5 * day)}} + f.reps = []BoxCertReport{{CustomerID: "old", ReportJSON: certReport(cert("gone.example.hu", f.now.Add(-2*day)))}} + f.step(0) + if len(f.events) != 1 || !strings.Contains(f.events[0], "„soon”") { + t.Fatalf("the first run: %v — want only the token that ends in 5 days", f.events) + } + f.step(2 * time.Hour) + f.step(2 * time.Hour) + if len(f.events) != 1 { + t.Fatalf("the already-ended things were told at a later run: %v", f.events[1:]) + } + // after the first run, a token that ends is told as ended (the normal rule) + f.tokens = append(f.tokens, EndingToken{ID: 3, Name: "third", ExpiresAt: f.now.Add(-time.Hour)}) + f.step(2 * time.Hour) + if len(f.events) != 2 || !strings.Contains(f.events[1], "„third” ENDED") { + t.Fatalf("a token that ended after the first run: %v", f.events[1:]) + } + + // a box that sends ANOTHER end date at every report + g := newEndDateFix() + g.step(0) // the baseline run + for i := 0; i < 48; i++ { + g.reps = []BoxCertReport{{CustomerID: "evil", ReportJSON: certReport(cert("x.example.hu", g.now.Add(5*day+time.Duration(i)*time.Minute)))}} + g.step(61 * time.Minute) + } + if len(g.events) > 3 { + t.Fatalf("a box that moves its end date by a minute each hour got %d mails in two days (want at most a few: a new mail needs a date a day later)", len(g.events)) + } + marksBefore := len(g.state) + // and a box that sends eight NEW names every hour + for i := 0; i < 24; i++ { + var cs []string + for n := 0; n < 8; n++ { + cs = append(cs, cert(fmt.Sprintf("h%dn%d.example.hu", i, n), g.now.Add(6*day))) + } + g.reps = []BoxCertReport{{CustomerID: "evil", ReportJSON: certReport(cs...)}} + g.step(61 * time.Minute) + } + if grew := len(g.state) - marksBefore; grew > webCertMaxMarks { + t.Fatalf("192 invented names made %d marks, want at most %d for one household", grew, webCertMaxMarks) + } +} diff --git a/hub/internal/monitor/signed_job_waits.go b/hub/internal/monitor/signed_job_waits.go index 83d99945..8fa376df 100644 --- a/hub/internal/monitor/signed_job_waits.go +++ b/hub/internal/monitor/signed_job_waits.go @@ -65,10 +65,10 @@ func (c *SignedJobWaitChecker) Check() { _ = c.setTold(key, now) waited := now.Sub(j.CreatedAt).Round(time.Minute) c.logger.Printf("[WARN] signed job %s for host %s was not taken (in the queue for %s)", j.JobID, j.HostID, waited) - msg := fmt.Sprintf("The signed job %s for %s is still in the queue after %s: the box did not take it. A box asks for its jobs every 15 minutes, so the box is off, cannot reach the hub, or its agent does not run. The job's validity has probably run out. Do: read why the box does not report, then sign a new job.", + msg := fmt.Sprintf("The signed job %s for %s is still in the queue after %s: the box did not take it. A box asks for its jobs every 15 minutes, so one of these is true: the box is off, it cannot reach the hub, its agent does not run — or its agent is too old to know this kind of job and leaves it in the queue on purpose. The job's validity has probably run out. Do: read the box's last report and its agent version, then sign a new job.", j.JobID, j.HostID, waited) details, _ := json.Marshal(map[string]any{"host_id": j.HostID, "job_id": j.JobID, - "queued_at": j.CreatedAt.UTC().Format(time.RFC3339), "waited_minutes": int(waited / time.Minute)}) + "queued_at": j.CreatedAt.UTC().Format(time.RFC3339), "waited_minutes": int(waited / time.Minute), "subject": j.HostID + ":" + j.JobID}) if c.onEvent != nil { c.onEvent(c.customer(j.HostID), EventSignedJobNotTaken, "warning", msg, string(details), "hub") } diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index 333ca36e..adffb496 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -622,6 +622,32 @@ const ( // operatorCooldownFor returns the operator-mail cooldown for an event type. Pinned by // TestOperatorCooldown_NodeLivenessBypassesQuietHour. +// perSubjectCooldownEvents (review of 2026-11-14): events whose checker tells each THING once and keeps a durable +// mark — one token, one certificate, one job, one guest, one cause, one daily job. Their operator cooldown is keyed +// per thing (details `subject`). With the plain `customer:type` key the second thing of the same kind inside the +// hour was suppressed — and, because its mark was already set, never told again: two tokens that end the same day, +// two daily checks skipped by one outage, a box that refuses every job. A named allow-list, like perAppCooldownEvents: +// every other type's key is byte-identical. Pinned by TestSubjectCooldown_*. +var perSubjectCooldownEvents = map[string]bool{ + "operator_token_ends_soon": true, "web_cert_ends_soon": true, "hub_daily_job_skipped": true, + "signed_job_not_taken": true, "signed_job_refused": true, "guest_power_gave_up": true, "guest_power_ok": true, + "proxmox_trust_problem": true, +} + +// cooldownSubjectSuffix returns ":"+subject for an allow-listed type whose details carry a non-empty `subject`. +func cooldownSubjectSuffix(eventType, detailsJSON string) string { + if !perSubjectCooldownEvents[eventType] || !strings.Contains(detailsJSON, "\"subject\"") { + return "" + } + var d struct { + Subject string `json:"subject"` + } + if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.Subject == "" { + return "" + } + return ":" + d.Subject +} + func operatorCooldownFor(eventType string) time.Duration { if nodeLivenessEvents[eventType] { return nodeLivenessDedupeWindow @@ -641,7 +667,8 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d // is byte-identical to v0.107.0's — pinned by TestR389_NoOtherEventTypeKeyChanges. cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON) + - cooldownStackSuffix(eventType, detailsJSON) + cooldownStorageSuffix(eventType, detailsJSON) + cooldownStackSuffix(eventType, detailsJSON) + cooldownStorageSuffix(eventType, detailsJSON) + + cooldownSubjectSuffix(eventType, detailsJSON) window := operatorCooldownFor(eventType) // R-723 (v0.126.0): a whole-guest tier skipped for absent storage in a box's FIRST HOUR is provisioning // still in flight, not a fault (measured 2026-09-29: the first local backup ran 7 min after enrolment, diff --git a/hub/internal/notify/lost_operator_mail.go b/hub/internal/notify/lost_operator_mail.go index 85439f95..162d322e 100644 --- a/hub/internal/notify/lost_operator_mail.go +++ b/hub/internal/notify/lost_operator_mail.go @@ -24,17 +24,20 @@ const lostMailLookBack = 48 * time.Hour // leaves an event row at once — see retryOperatorEmail). And this summary is an operator mail like any other: when the // mail service is still down it is retried three times and can be lost too — the next start then lists it. // Returns the number of lost mails told. Pinned by TestLostOperatorMail_*. -func (d *Dispatcher) ReportLostOperatorMails(now time.Time) int { +// +// processStart is when THIS hub process began (cmd/hub: a package-level time). Only mails that failed before it are +// lost: one that failed since may still be retried by this process's own timers. +func (d *Dispatcher) ReportLostOperatorMails(now, processStart time.Time) int { since := now.Add(-lostMailLookBack) if mark := d.store.OSAlarmRaised("operator_mail_lost_scan"); mark.After(since) { since = mark } - lost, err := d.store.LostOperatorMails(since) + lost, err := d.store.LostOperatorMails(since, processStart) if err != nil { d.logger.Printf("[WARN] lost operator mail scan failed: %v", err) return 0 } - if err := d.store.SetOSAlarmRaised("operator_mail_lost_scan", now); err != nil { + if err := d.store.SetOSAlarmRaised("operator_mail_lost_scan", processStart); err != nil { d.logger.Printf("[WARN] lost operator mail scan: could not record the scan time: %v", err) } if len(lost) == 0 { @@ -53,8 +56,8 @@ func (d *Dispatcher) ReportLostOperatorMails(now time.Time) int { break } msg := m.Message - if len(msg) > 400 { - msg = msg[:400] + "…" + if r := []rune(msg); len(r) > 400 { // by rune: the texts hold „ ” — and a byte cut would split one + msg = string(r[:400]) + "…" } cust := m.CustomerID if cust == "" { diff --git a/hub/internal/notify/lost_operator_mail_test.go b/hub/internal/notify/lost_operator_mail_test.go index afd972b3..465bc42a 100644 --- a/hub/internal/notify/lost_operator_mail_test.go +++ b/hub/internal/notify/lost_operator_mail_test.go @@ -65,8 +65,9 @@ func TestLostOperatorMail_ToldAtTheNextStart(t *testing.T) { got = append(got, sentMail{to, subject, body}) return nil } - now := time.Now().Add(time.Minute) - if n := d2.ReportLostOperatorMails(now); n != 1 { + now := time.Now().Add(2 * time.Second) + time.Sleep(1100 * time.Millisecond) // the log's times have one-second grain: the "restart" is a second later + if n := d2.ReportLostOperatorMails(now, now); n != 1 { t.Fatalf("lost mails told = %d, want 1 (the one whose retries the restart ate; not the one that went out on a retry)", n) } if len(got) != 1 || got[0].to != "op@felhom.eu" { @@ -85,7 +86,7 @@ func TestLostOperatorMail_ToldAtTheNextStart(t *testing.T) { t.Fatalf("event rows = %+v, want one operator_mail_lost row", evs) } // a second restart: the same mail is not told again - if n := d2.ReportLostOperatorMails(now.Add(time.Minute)); n != 0 || len(got) != 1 { + if n := d2.ReportLostOperatorMails(now.Add(time.Minute), now.Add(time.Minute)); n != 0 || len(got) != 1 { t.Fatalf("a second start told %d again (mails now %d)", n, len(got)) } _ = sent @@ -94,6 +95,27 @@ func TestLostOperatorMail_ToldAtTheNextStart(t *testing.T) { } } +// Review of 2026-11-14: „at a start no timer is alive” is not true for a mail that failed in THIS process (the scan +// runs some way into the start; a daily-check alarm or a box's event can already have failed and be waiting for its +// retry). Such a mail must not be told as „never sent” — it may still arrive. +func TestLostOperatorMail_AMailThatFailedInThisProcessIsNotLost(t *testing.T) { + d, _, down := lostMailDispatcher(t) + processStart := time.Now().Add(-2 * time.Second) // this process started two seconds ago + *down = true + d.afterFn = func(time.Duration, func()) {} // its retry timer is alive and has not fired yet + d.processOperator("c1", "whole_guest_backup_failed", "error", "Whole-guest backup FAILED on demo-hp", "", "box") + *down = false + if n := d.ReportLostOperatorMails(time.Now().Add(time.Second), processStart); n != 0 { + t.Fatalf("a mail that failed AFTER this process started was told as lost (%d)", n) + } + // the next process (a later start) does tell it + time.Sleep(1100 * time.Millisecond) + later := time.Now().Add(time.Second) + if n := d.ReportLostOperatorMails(later, later); n != 1 { + t.Fatalf("the next start must tell it: %d", n) + } +} + // The other half (clock list CH04-7): the hub keeps running and all three retries fail. A log line was the only // trace; now an event row says which mail was never sent. (No mail: the mail service is what failed.) func TestLostOperatorMail_GiveUpLeavesARow(t *testing.T) { diff --git a/hub/internal/notify/subject_cooldown_test.go b/hub/internal/notify/subject_cooldown_test.go new file mode 100644 index 00000000..c902610f --- /dev/null +++ b/hub/internal/notify/subject_cooldown_test.go @@ -0,0 +1,43 @@ +package notify + +import ( + "io" + "log" + "strings" + "testing" +) + +// Review of 2026-11-14: the checkers behind these events tell each THING once and keep a durable mark. With the +// plain `customer:type` cooldown key the second thing of the same kind inside the hour was suppressed — and never +// told again, because its mark was already set. Two tokens that end the same day must give TWO mails; the same +// token again inside the hour gives none; and a type that is not on the list keeps its old key, subject or not. +// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt). +func TestSubjectCooldown_TwoThingsTwoMails(t *testing.T) { + st := newDispStore(t) + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + var bodies []string + d.sendEmailFn = func(_, _, body string, _ map[string]string) error { bodies = append(bodies, body); return nil } + d.processOperator("", "operator_token_ends_soon", "warning", "token A ends", `{"subject":"token:1:soon"}`, "hub") + d.processOperator("", "operator_token_ends_soon", "warning", "token B ends", `{"subject":"token:2:soon"}`, "hub") + d.processOperator("", "operator_token_ends_soon", "warning", "token A ends", `{"subject":"token:1:soon"}`, "hub") + if len(bodies) != 2 || !strings.Contains(bodies[0], "token A") || !strings.Contains(bodies[1], "token B") { + t.Fatalf("mails = %d, want two (one per token) and none for the repeat: %q", len(bodies), bodies) + } + // every type on the list has the suffix; a type that is not on it has none, also with a subject in its details + for typ := range perSubjectCooldownEvents { + if cooldownSubjectSuffix(typ, `{"subject":"x"}`) != ":x" { + t.Errorf("%s: no per-subject key", typ) + } + if !operatorOnlyEvents[typ] { + t.Errorf("%s is keyed per subject but is not operator-only", typ) + } + } + for _, typ := range []string{"backup_failed", "app_start_failed", "tunnel_down", "box_clock_wrong"} { + if s := cooldownSubjectSuffix(typ, `{"subject":"x"}`); s != "" { + t.Errorf("%s: its cooldown key changed (%q) — only the listed types may", typ, s) + } + } + if cooldownSubjectSuffix("signed_job_refused", `{"other":1}`) != "" || cooldownSubjectSuffix("signed_job_refused", `not json "subject"`) != "" { + t.Error("no subject, or details that are not JSON, must give no suffix") + } +} diff --git a/hub/internal/offsite/account_record_test.go b/hub/internal/offsite/account_record_test.go index b78133b6..dfa0191d 100644 --- a/hub/internal/offsite/account_record_test.go +++ b/hub/internal/offsite/account_record_test.go @@ -27,6 +27,11 @@ func TestKeepAccountRecord_FourCases(t *testing.T) { {"a config that carries BOTH a ticked tier and a record: the record goes (the descriptor is the record)", withRecord, `{"offsite":{"enabled":true,"type":"shared","user":"u1-sub3"},"offsite_account":{"type":"shared","user":"u1-sub3"}}`, "", "", `"offsite"`}, {"ticked, never provisioned, then unticked: no account was made, no record", choiceOnly, unticked, "", "", `"git"`}, + // review 2026-11-14: a tick with the OTHER type makes a second account; the first one still exists + {"unticked shared, then ticked as DEDICATED: the shared account's record stays", withRecord, + `{"offsite":{"enabled":true,"type":"dedicated","user":"u9","host":"u9.example"}}`, "u1-sub3", "2026-11-01T00:00:00Z", `"offsite"`}, + {"switched from shared to dedicated in ONE save: the shared account gets a record", provisioned, + `{"offsite":{"enabled":true,"type":"dedicated","user":"u9","host":"u9.example"}}`, "u1-sub3", "2026-11-14T20:00:00Z", `"offsite"`}, {"never ticked: nothing", `{}`, unticked, "", "", `"git"`}, {"an empty previous config", ``, `{}`, "", "", ``}, } { diff --git a/hub/internal/offsite/offsite.go b/hub/internal/offsite/offsite.go index 1dbe37e0..7810280f 100644 --- a/hub/internal/offsite/offsite.go +++ b/hub/internal/offsite/offsite.go @@ -514,8 +514,6 @@ func ClearProvisionedDescriptor(configJSON string) (string, error) { if err != nil { return "", err } - // D71: the record of an unticked tier's account dies with the account too (RESET took it down before this leg). - configJSON = dropAccountRecord(configJSON) if cur == nil { return configJSON, nil // no offsite block — nothing to clear } @@ -523,6 +521,11 @@ func ClearProvisionedDescriptor(configJSON string) (string, error) { return MergeDescriptor(configJSON, cleared) } +// DropAccountRecord removes the record. Called by the RESET only AFTER the provider leg really took the account +// down — never because a leg was skipped (no provider configured on this hub): then the account still exists and its +// record must stay (review, 2026-11-14). +func DropAccountRecord(configJSON string) string { return dropAccountRecord(configJSON) } + func dropAccountRecord(configJSON string) string { obj := map[string]json.RawMessage{} if json.Unmarshal([]byte(configJSON), &obj) != nil { @@ -573,7 +576,9 @@ func ReadAccountRecord(configJSON string) *AccountRecord { // KeepAccountRecord is called on every save of a household's config, with the config as it WAS and as the form // built it. It returns the new config with the account record carried: -// - the tier is ticked in the new config: no record (the descriptor itself is the record); +// - the tier is ticked in the new config with the SAME type as the account before: no record (the same account +// is back in use; the descriptor itself is the record). Ticked with the OTHER type: the record of the first +// account stays — two accounts exist; // - the tier was ticked AND PROVISIONED before and is unticked now: a record is written from the old descriptor; // - a record was there before and the tier is still unticked: the record is carried (the form does not know it); // - else: nothing — a tier that was ticked and never provisioned made no account. @@ -592,18 +597,26 @@ func KeepAccountRecord(prevJSON, newJSON string, now time.Time) (string, error) if err != nil { return "", err } - if newD != nil && newD.Enabled { - delete(obj, AccountRecordKey) - } else { - var rec *AccountRecord - if prevD, _ := ReadDescriptor(prevJSON); prevD != nil && prevD.Enabled && (prevD.User != "" || prevD.Host != "") { - rec = &AccountRecord{Type: prevD.Type, User: prevD.User, Host: prevD.Host, UntickedAt: now.UTC().Format(time.RFC3339)} - } else if prevRec := ReadAccountRecord(prevJSON); prevRec != nil { - rec = prevRec - } - if rec == nil { + // The account that existed before this save and is NOT the one the new config uses. + var rec *AccountRecord + if prevD, _ := ReadDescriptor(prevJSON); prevD != nil && prevD.Enabled && (prevD.User != "" || prevD.Host != "") { + rec = &AccountRecord{Type: prevD.Type, User: prevD.User, Host: prevD.Host, UntickedAt: now.UTC().Format(time.RFC3339)} + } else if prevRec := ReadAccountRecord(prevJSON); prevRec != nil { + rec = prevRec + } + if newD != nil && newD.Enabled && rec != nil && rec.Type == newD.Type { + // ticked again with the SAME type: the provider is asked by the household's label, so the same account is back + // in use and the descriptor is its record. A tick with the OTHER type makes a second account — the first one + // still exists, so its record STAYS (review, 2026-11-14: it was dropped, and a later RESET then took down only + // the new account). + rec = nil + } + if rec == nil { + if _, had := obj[AccountRecordKey]; !had { return newJSON, nil } + delete(obj, AccountRecordKey) + } else { rb, _ := json.Marshal(rec) obj[AccountRecordKey] = rb } diff --git a/hub/internal/store/customer_delete.go b/hub/internal/store/customer_delete.go index fb72d4eb..4e917ddf 100644 --- a/hub/internal/store/customer_delete.go +++ b/hub/internal/store/customer_delete.go @@ -116,6 +116,12 @@ func (s *Store) PurgeCustomerResidue(customerID string) error { return fmt.Errorf("PurgeCustomerResidue %s: purge %s: %w", customerID, q.table, err) } } + // D73: the hub's own alarm marks that name the HOUSEHOLD (today: the web-certificate end dates, keyed + // `os_alarm:end_date:web_cert:::`). substr, not LIKE: an id may hold `_` or `%`. + markPrefix := "os_alarm:end_date:web_cert:" + customerID + ":" + if _, err := tx.Exec(`DELETE FROM hub_settings WHERE substr(key, 1, ?) = ?`, len(markPrefix), markPrefix); err != nil { + return fmt.Errorf("PurgeCustomerResidue %s: alarm marks: %w", customerID, err) + } // R-244: de-reference (not delete) the customer in the shared app-log issue rows. if _, _, err := dereferenceCustomerFromAppIssues(tx, customerID); err != nil { return fmt.Errorf("PurgeCustomerResidue %s: app_log_issues: %w", customerID, err) diff --git a/hub/internal/store/d73_gone_rows_test.go b/hub/internal/store/d73_gone_rows_test.go index c43f87a1..3507d12d 100644 --- a/hub/internal/store/d73_gone_rows_test.go +++ b/hub/internal/store/d73_gone_rows_test.go @@ -132,6 +132,11 @@ func TestD73_PurgeRemovesTheHouseholdsRows_JournalStays(t *testing.T) { if _, err := s.db.Exec(`INSERT INTO host_deletions (host_id, customer_id, escrow_acked) VALUES ('box-cust-a', 'cust-a', 0)`); err != nil { t.Fatal(err) } + for _, cust := range []string{"cust-a", "cust-b", "cust-a2"} { + if err := s.SetOSAlarmRaised("end_date:web_cert:"+cust+":x.example.hu:soon", time.Now()); err != nil { + t.Fatal(err) + } + } res, err := s.CustomerResidue("cust-a") if err != nil || res.OffsiteKeys != 1 || res.OffsiteWindows != 1 || res.OffsiteAbandonRequests != 1 || res.Total() < 3 { t.Fatalf("the preview's residue does not count the three tables: %+v (%v)", res, err) @@ -156,6 +161,13 @@ func TestD73_PurgeRemovesTheHouseholdsRows_JournalStays(t *testing.T) { t.Errorf("journal %s: %d row(s), want 1 — the delete journal stays (the ruling's one exception)", j, n) } } + // the hub's own alarm marks that name the HOUSEHOLD went too (review 2026-11-14) — and only that household's, + // also when another household's id starts the same way + for cust, want := range map[string]int{"cust-a": 0, "cust-b": 1, "cust-a2": 1} { + if n := len(s.OSAlarmKeys("end_date:web_cert:" + cust + ":")); n != want { + t.Errorf("web-certificate marks of %s: %d, want %d", cust, n, want) + } + } // the action log goes with the audit rows: not at 100 days, gone after a year now := time.Now().UTC() if _, _, err := s.PruneDeletedCustomerAudit(now.Add(100*24*time.Hour), DeletedCustomerAuditKeep); err != nil { @@ -223,3 +235,27 @@ func TestD73_PurgeOrphanRows_OnlyTheGone(t *testing.T) { t.Errorf("a second sweep removed %v — it must find nothing", again) } } + +// Review of 2026-11-14: a cleared alarm mark must leave NO row. It used to leave an empty one for ever, and several +// marks carry a name a box sends (a guest number, a job id, a certificate name) — empty rows without limit. +func TestOSAlarm_AClearedMarkLeavesNoRow(t *testing.T) { + s := newTestStore(t) + if err := s.SetOSAlarmRaised("probe:x", time.Now()); err != nil { + t.Fatal(err) + } + if s.OSAlarmRaised("probe:x").IsZero() || d73Count(t, s, `SELECT COUNT(*) FROM hub_settings WHERE key = 'os_alarm:probe:x'`) != 1 { + t.Fatal("setup: the mark was not set") + } + if err := s.SetOSAlarmRaised("probe:x", time.Time{}); err != nil { + t.Fatal(err) + } + if n := d73Count(t, s, `SELECT COUNT(*) FROM hub_settings WHERE key = 'os_alarm:probe:x'`); n != 0 { + t.Fatalf("a cleared mark left %d row(s)", n) + } + if !s.OSAlarmRaised("probe:x").IsZero() { + t.Fatal("a cleared mark still reads as raised") + } + if err := s.SetOSAlarmRaised("probe:never-set", time.Time{}); err != nil { + t.Fatalf("clearing a mark that was never set: %v", err) + } +} diff --git a/hub/internal/store/os_updates.go b/hub/internal/store/os_updates.go index a321c984..7153de89 100644 --- a/hub/internal/store/os_updates.go +++ b/hub/internal/store/os_updates.go @@ -404,11 +404,32 @@ func (s *Store) OSAlarmRaised(key string) time.Time { // SetOSAlarmRaised records (t non-zero) or clears (zero t) an OS alarm. func (s *Store) SetOSAlarmRaised(key string, t time.Time) error { - v := "" - if !t.IsZero() { - v = t.UTC().Format(time.RFC3339) + if t.IsZero() { + // a cleared alarm leaves NO row (it used to leave an empty one for ever; several keys carry a name a box + // sends, so empty rows could be made without limit). A missing row reads as "not raised", as "" did. + _, err := s.db.Exec(`DELETE FROM hub_settings WHERE key = ?`, "os_alarm:"+key) + return err } - return s.setSetting("os_alarm:"+key, v) + return s.setSetting("os_alarm:"+key, t.UTC().Format(time.RFC3339)) +} + +// OSAlarmKeys lists the raised alarm marks whose key starts with prefix (the keys as OSAlarmRaised takes them). The +// prefix is matched with substr, not LIKE: a host or customer id may hold `_` or `%`. A read error is no keys. +func (s *Store) OSAlarmKeys(prefix string) []string { + full := "os_alarm:" + prefix + rows, err := s.db.Query(`SELECT key FROM hub_settings WHERE substr(key, 1, ?) = ? AND value != ''`, len(full), full) + if err != nil { + return nil + } + defer rows.Close() + var out []string + for rows.Next() { + var k string + if rows.Scan(&k) == nil { + out = append(out, strings.TrimPrefix(k, "os_alarm:")) + } + } + return out } // HostAgentVersionsSeen returns, per agent version a host ever reported, the first time it was seen (the stale-leg diff --git a/hub/internal/store/silent_jobs.go b/hub/internal/store/silent_jobs.go index 35fb0aec..f1447341 100644 --- a/hub/internal/store/silent_jobs.go +++ b/hub/internal/store/silent_jobs.go @@ -41,17 +41,18 @@ type LostOperatorMail struct { } // LostOperatorMails returns the operator mails recorded after `since` whose newest delivery record (sent, failed or -// held — a cooldown's `suppressed` row is not a delivery) is `failed`: the mail was never sent. Read at a hub START, -// when no retry timer can be alive, so every such row is a mail nothing will send (R-1383; clock list CH04-4, CH04-7). -func (s *Store) LostOperatorMails(since time.Time) ([]LostOperatorMail, error) { +// held — a cooldown's `suppressed` row is not a delivery) is `failed`: the mail was never sent. Only rows made BEFORE +// `before` — the moment this hub process started: a mail that failed in THIS process still has its retry timers, and +// telling it as „never sent” and then sending it would be a false alarm (review, 2026-11-14). (R-1383; CH04-4, CH04-7.) +func (s *Store) LostOperatorMails(since, before time.Time) ([]LostOperatorMail, error) { rows, err := s.db.Query(` SELECT n.customer_id, n.event_type, n.severity, n.message, COALESCE(n.error_message, ''), n.created_at FROM notification_log n - WHERE n.channel = 'operator' AND n.status = 'failed' AND n.created_at > ? + WHERE n.channel = 'operator' AND n.status = 'failed' AND n.created_at > ? AND n.created_at < ? AND n.id = (SELECT MAX(m.id) FROM notification_log m WHERE m.channel = 'operator' AND m.customer_id = n.customer_id AND m.event_type = n.event_type AND m.message = n.message AND m.status IN ('sent', 'failed', 'held')) - ORDER BY n.id ASC`, since.UTC().Format("2006-01-02 15:04:05")) + ORDER BY n.id ASC`, since.UTC().Format("2006-01-02 15:04:05"), before.UTC().Format("2006-01-02 15:04:05")) if err != nil { return nil, err } diff --git a/hub/internal/web/customer_reset.go b/hub/internal/web/customer_reset.go index 78bfd2f6..7959e347 100644 --- a/hub/internal/web/customer_reset.go +++ b/hub/internal/web/customer_reset.go @@ -44,6 +44,16 @@ func offsiteChoice(configJSON string) (enabled bool, typ string) { return o.Offsite.Enabled, o.Offsite.Type } +// offsiteAccountTypes lists the provider account types a RESET must take down: `first` (what offsiteChoice returned) +// and the kept record's type when it is another one. Each once. +func offsiteAccountTypes(configJSON, first string) []string { + types := []string{first} + if rec := offsite.ReadAccountRecord(configJSON); rec != nil && rec.Type != first { + types = append(types, rec.Type) + } + return types +} + // handleCustomerResetPreview — GET /configs/{id}/reset. Returns the live inventory the confirm surface // renders (ruling 4): what a RESET would destroy right now. Read-only — no writes, no external calls // beyond the label lookups needed to name the offsite resource. `refused` is true when a host row still @@ -216,6 +226,7 @@ func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerCon } // Leg: Hetzner offsite (repo DATA destroyed). Only when the customer chose an offsite tier. + hetznerDone := false // the provider leg really ran to its end (not skipped) — only then the account's record may go offsiteEnabled, offsiteType := false, "" if cfg != nil { offsiteEnabled, offsiteType = offsiteChoice(cfg.ConfigJSON) @@ -225,12 +236,17 @@ func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerCon s.logger.Printf("[WARN] %s: no config row — the offsite (Hetzner) teardown CANNOT be determined and is SKIPPED; verify the Hetzner side by hand", customerID) _ = s.store.UpdateResetLeg(resetID, "hetzner", "skipped_no_config") } else if offsiteEnabled && s.offsite != nil { - if derr := s.offsite.Deprovision(ctx, customerID, offsiteType); derr != nil { - _ = s.store.UpdateResetLeg(resetID, "hetzner", "failed") - s.logger.Printf("[ERROR] reset %s: hetzner deprovision FAILED (journal #%d retained; re-run to resume): %v", customerID, resetID, derr) - return &resetLegError{Leg: "hetzner", Status: http.StatusBadGateway, Err: derr, - Msg: "Reset incomplete: the offsite (Hetzner) teardown failed — nothing was purged; re-run to resume. (" + derr.Error() + ")"} + // Every account the config knows: the ticked tier's, and — when the tier was unticked, or ticked again with + // the other type — the one in the kept record. Each type once. + for _, typ := range offsiteAccountTypes(cfg.ConfigJSON, offsiteType) { + if derr := s.offsite.Deprovision(ctx, customerID, typ); derr != nil { + _ = s.store.UpdateResetLeg(resetID, "hetzner", "failed") + s.logger.Printf("[ERROR] reset %s: hetzner deprovision (%s) FAILED (journal #%d retained; re-run to resume): %v", customerID, typ, resetID, derr) + return &resetLegError{Leg: "hetzner", Status: http.StatusBadGateway, Err: derr, + Msg: "Reset incomplete: the offsite (Hetzner) teardown failed — nothing was purged; re-run to resume. (" + derr.Error() + ")"} + } } + hetznerDone = true _ = s.store.UpdateResetLeg(resetID, "hetzner", "ok") s.logger.Printf("[INFO] reset %s: offsite deprovisioned (repo data destroyed)", customerID) } else { @@ -275,6 +291,9 @@ func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerCon if cfg == nil { _ = s.store.UpdateResetLeg(resetID, "descriptor", "skipped_no_config") } else { + if hetznerDone { + cfg.ConfigJSON = offsite.DropAccountRecord(cfg.ConfigJSON) // D71: the record goes with the account + } newConfigJSON, cerr := offsite.ClearProvisionedDescriptor(cfg.ConfigJSON) if cerr != nil { _ = s.store.UpdateResetLeg(resetID, "descriptor", "failed") diff --git a/hub/internal/web/d71_offsite_account_record_test.go b/hub/internal/web/d71_offsite_account_record_test.go index b683762d..37f5f823 100644 --- a/hub/internal/web/d71_offsite_account_record_test.go +++ b/hub/internal/web/d71_offsite_account_record_test.go @@ -172,3 +172,34 @@ func TestD71_NeverTickedMakesNoRecord(t *testing.T) { } _ = http.StatusOK } + +// Review of 2026-11-14: (1) a RESET on a hub with NO provider configured skips the provider leg — the account still +// exists, so its record must stay (it was dropped with the descriptor). (2) Unticked as shared and ticked again as +// dedicated: two accounts exist; the RESET takes BOTH down. +func TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo(t *testing.T) { + s, st, _, _ := d71Server(t) + d71Save(t, s, d71Tick) + d71Save(t, s, "") + s.offsite = nil // this hub has no provider token + if rr := postReset(t, s, "acme", url.Values{"confirm_id": {"acme"}}); rr.Code != http.StatusSeeOther { + t.Fatalf("reset: %d %s", rr.Code, rr.Body.String()) + } + cfg, _ := st.GetCustomerConfig("acme") + cr, _ := st.LatestCustomerReset("acme") + if cr == nil || cr.Legs["hetzner"] != "skipped" { + t.Fatalf("journal: %+v", cr) + } + if offsite.ReadAccountRecord(cfg.ConfigJSON) == nil { + t.Fatalf("the provider leg was SKIPPED and the account's record is gone: %s", cfg.ConfigJSON) + } + + if got := offsiteAccountTypes(`{"offsite":{"enabled":true,"type":"dedicated"},"offsite_account":{"type":"shared","user":"u1"}}`, "dedicated"); len(got) != 2 || got[0] != "dedicated" || got[1] != "shared" { + t.Fatalf("a household with a dedicated tier AND a kept shared record: the RESET would take down %v, want both", got) + } + if got := offsiteAccountTypes(`{"offsite":{"enabled":true,"type":"shared"}}`, "shared"); len(got) != 1 { + t.Fatalf("one account: %v", got) + } + if got := offsiteAccountTypes(`{"offsite_account":{"type":"shared","user":"u1"}}`, "shared"); len(got) != 1 { + t.Fatalf("an unticked tier with its record is ONE account, not two: %v", got) + } +} diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index b70a5ad7..68b381fd 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,11 @@ ## Unreleased (2026-11-14 night) — the installer: a new box trusts Proxmox by a copy of its own root CA (R-1380, D80) — on `main`, NOT tagged, so NOT published +- **Installer, after review (2026-11-14, not tagged):** `resolve_pve_trust` under `set -euo pipefail` — the installer's + own mode. A failed pipeline inside `fp=$(…)` ended the whole installer when Proxmox did not answer; now it leaves + the fingerprint empty and goes on (test `test_pve_trust_no_answer_from_proxmox_does_not_end_the_installer`, seen + red). The proof reads openssl's answer whole and searches it after, in place of a pipe into `grep -q` (reasoned by + the reviewer as a way to read a good copy as „not verified”; NOT reproduced). The trust tests now run the function + under strict mode. - `felhom-host-install.sh` `resolve_pve_trust` (new, called by `step_agent_config`): copy `/etc/pve/pve-root-ca.pem` to `/etc/felhom-agent/pve-root-ca.pem` when it is one unexpired CA certificate; prove a TLS handshake to `127.0.0.1:8006` with the copy alone; then write `proxmox.tls.ca_file` and NO fingerprint. Anything else: the served diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index 3cf67580..eeb7de01 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -3188,7 +3188,8 @@ resolve_pve_trust() { local addr="${PVE_API_ADDR:-127.0.0.1:8006}" fp why="" PVE_CA_FILE=""; PVE_TLS_FP="" # the SERVED leaf cert fingerprint (not pvesh node info — may differ): the fallback - fp=$(echo | openssl s_client -connect "$addr" 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//') + # `|| fp=""`: under `set -euo pipefail` a failed pipeline inside the substitution would END the installer here. + fp=$(echo | openssl s_client -connect "$addr" 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//') || fp="" if [[ ! -r "$src" ]]; then why="$src is not readable" elif [[ "$(grep -c 'BEGIN CERTIFICATE' "$src")" != "1" ]] || grep -q 'PRIVATE KEY' "$src"; then @@ -3207,8 +3208,12 @@ resolve_pve_trust() { if install -m 0644 "$src" "$dst.felhom-new" && mv -f "$dst.felhom-new" "$dst"; then # THE PROOF, before the config names the copy: verified by the copy ALONE (no system roots), for the address # the agent calls (the IP must be in the certificate, as the agent's own check demands) - if echo | openssl s_client -connect "$addr" -CAfile "$dst" -no-CApath -verify_return_error \ - -verify_ip "${addr%%:*}" 2>&1 | grep -q 'Verify return code: 0 (ok)'; then + # The answer is read WHOLE into a variable and searched after: `… | grep -q` leaves at its first match while + # s_client still writes, and under `pipefail` that broken pipe would read as "did not verify" for a good copy. + local proof + proof=$(echo | openssl s_client -connect "$addr" -CAfile "$dst" -no-CApath -verify_return_error \ + -verify_ip "${addr%%:*}" 2>&1) || true + if grep -q 'Verify return code: 0 (ok)' <<<"$proof"; then PVE_CA_FILE="$dst" log_info " Proxmox trust: the box's own root CA ($dst; a handshake to $addr verified with it alone) — no pinned certificate" return 0 diff --git a/scripts/test_hostinstall.py b/scripts/test_hostinstall.py index dd64d7a8..3fc2e998 100644 --- a/scripts/test_hostinstall.py +++ b/scripts/test_hostinstall.py @@ -763,8 +763,10 @@ class _PVE: self.proc.kill(); self.proc.wait() -def _trust_run(source, dry=False): - """Run resolve_pve_trust against a real TLS server. Returns (ca_file, fp, copy bytes or None, source bytes, output, leaf fp, dst).""" +def _trust_run(source, dry=False, dead_port=False): + """Run resolve_pve_trust against a real TLS server — under `set -euo pipefail`, as the installer itself runs (line + 195 of the script): a function that only works without strict mode proves nothing about the installer. + Returns (ca_file, fp, copy bytes or None, source bytes, output, leaf fp, dst).""" if not shutil.which("openssl"): return None sb = Sandbox() @@ -772,9 +774,10 @@ def _trust_run(source, dry=False): try: dst = sb.path("etc-felhom-agent", "pve-root-ca.pem") src = os.path.join(pve.dir, source) - rc, out = sb.run(prelude(dry) + func("resolve_pve_trust") + + port = 1 if dead_port else pve.port # port 1: nothing listens → the handshake cannot even start + rc, out = sb.run(prelude(dry) + func("resolve_pve_trust") + 'set -euo pipefail\n' 'PVE_CA_SOURCE="%s" PVE_CA_COPY="%s" PVE_API_ADDR=127.0.0.1:%d\nresolve_pve_trust\n' - 'echo "RESULT ca=[$PVE_CA_FILE] fp=[$PVE_TLS_FP]"\n' % (src, dst, pve.port)) + 'echo "RESULT ca=[$PVE_CA_FILE] fp=[$PVE_TLS_FP]"\n' % (src, dst, port)) assert rc == 0, out m = re.search(r"RESULT ca=\[(.*?)\] fp=\[(.*?)\]", out) assert m, out @@ -814,6 +817,18 @@ def test_pve_trust_a_source_that_is_not_a_ca_is_never_copied(): assert why in out, "%s: the reason is not named:\n%s" % (source, out) +def test_pve_trust_no_answer_from_proxmox_does_not_end_the_installer(): + """Found by review 2026-11-14: under `set -euo pipefail` a failed pipeline inside `fp=$(…)` ended the whole installer at + that line. Proxmox not answering (it restarts) must leave the function running to its end: no CA file (no proof), no + fingerprint, and the two warnings — the run reaches RESULT.""" + r = _trust_run("ca.pem", dead_port=True) + if r is None: + print(" (skipped: no openssl here)"); return + ca, fp, _, _, out, _, _ = r + assert ca == "" and fp == "", "nothing answered, so nothing can be proven or pinned. Got ca=%r fp=%r\n%s" % (ca, fp, out) + assert "did not verify" in out and "could not compute TLS fingerprint" in out, out + + def test_pve_trust_dry_run_writes_nothing(): r = _trust_run("ca.pem", dry=True) if r is None: