diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 30f072bd..f2dfaf09 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -1007,7 +1007,11 @@ a config that holds both is a pinned box — the agent treats it as „not moved certificates.** The root step writes the new root FIRST and keeps the copy's previous first certificate behind it: the copy is replaced before anything can be proven with the new root, and Proxmox serves its old certificate until pveproxy is reloaded — without the old root, an agent that restarted in that window had no root that verified the -served certificate and no fingerprint. Never more than two; an ended one verifies nothing. (b) **One run at a time** +served certificate and no fingerprint. **Only for the changeover:** at every later run the step makes one handshake to +the local Proxmox (`127.0.0.1:8006`) with the new root alone and drops the previous root as soon as that verifies, or +when the previous root is no longer a CA that has not ended. The copy is compared exactly, never by its start. By +hand, to drop an old root at once (a root that was changed because it leaked): delete the copy; the next run writes +the new root alone. Checked with the real openssl and a real TLS server in the wrapper's tests. (b) **One run at a time** (`Ensure` is called from the start, the timer, a failed collect and a bundle install), and the root step has a 90-second deadline — it runs on the daemon's start path and inside the report loop. (c) **One lock for every writer of `agent.json`** in the process (`config.FileMu`: the trust move, the PBS seed, the backup-target move): each reads diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 4e00da2e..f54d8f24 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -731,7 +731,8 @@ the same report lists (not running), 16 entries at most. **The mails (after review, 2026-11-14):** each of these events tells one thing once and keeps a mark, so its mail cooldown is keyed per thing (`notify.perSubjectCooldownEvents`) — with the plain per-type key the second job, guest or skipped check inside the hour was suppressed and never told. The lost-mail scan tells only mails that failed before -this hub process started (one that failed since still has its retry). One box holds at most 8 give-up marks. +this hub process started (one that failed since still has its retry). One box holds at most 8 give-up marks, and the +mark of a guest the box stops listing keeps its slot for a week. **Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red). diff --git a/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt b/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt index d71c0b32..02bf1e99 100644 --- a/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt +++ b/documentation/audits/night-build-2026-11-14/review/redproof-agent-review-fixes.txt @@ -67,3 +67,42 @@ the real file: 0 failing line(s) 3 mutations, 0 not a red-proof +---- the CA copy, second pass (after the commit's security review: the previous root only for the changeover; exact compare) ---- +== MUTATION: the previous root is never dropped (it stays a trust anchor for ever) + RED on the mutant: 3 failing line(s) + FAIL: test_the_previous_root_stays_only_for_the_changeover (__main__.PVECACopy.test_the_previous_root_stays_only_for_the_changeover) + AssertionError: b'---[86 chars]---\n-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJ[49 chars]--\n' != b'---[86 chars]---\n' : after the changeover the copy is the new root alone + FAILED (failures=1) +== MUTATION: the previous root is dropped while Proxmox still serves the old certificate + RED on the mutant: 3 failing line(s) + FAIL: test_the_previous_root_stays_only_for_the_changeover (__main__.PVECACopy.test_the_previous_root_stays_only_for_the_changeover) + AssertionError: Tuples differ: (0, 'written', [('/etc/felhom-agent/pve-root-c[108 chars]20)]) != (0, 'same', []) + FAILED (failures=1) +== MUTATION: a copy that only STARTS with the source is same (unchecked content stays) + RED on the mutant: 4 failing line(s) + KeyError: 'kept_previous' + FAIL: test_a_copy_with_anything_unchecked_in_it_is_rewritten (__main__.PVECACopy.test_a_copy_with_anything_unchecked_in_it_is_rewritten) + AssertionError: b'anything' unexpectedly found in b'-----BEGIN CERTIFICATE-----\nWllYV1ZVVFNSUVBPTk1MS0pJSEdGRURDQkE=\n-----END CERTIFICATE-----\nanything at all\n' : text after the source +== MUTATION: an ended previous root is kept at a change + RED on the mutant: 3 failing line(s) + FAIL: test_the_previous_root_stays_only_for_the_changeover (__main__.PVECACopy.test_the_previous_root_stays_only_for_the_changeover) + AssertionError: b'---[86 chars]---\n-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJ[49 chars]--\n' != b'---[86 chars]---\n' + FAILED (failures=1) +== MUTATION: the handshake is verified with the system's roots too + RED on the mutant: 3 failing line(s) + FAIL: test_the_previous_root_stays_only_for_the_changeover (__main__.PVECACopy.test_the_previous_root_stays_only_for_the_changeover) + AssertionError: '-no-CApath' not found in ['openssl', 's_client', '-connect', '127.0.0.1:8006', '-CAfile', '/etc/pve/pve-root-ca.pem', '-verify_return_error'] + FAILED (failures=1) +== MUTATION: any answer of the handshake counts as verified + RED on the mutant: 5 failing line(s) + FAIL: test_the_previous_root_stays_only_for_the_changeover (__main__.PVECACopy.test_the_previous_root_stays_only_for_the_changeover) + AssertionError: Tuples differ: (0, 'written', [('/etc/felhom-agent/pve-root-c[108 chars]20)]) != (0, 'same', []) + FAIL: test_with_the_real_openssl_and_the_real_runner (__main__.PVECACopy.test_with_the_real_openssl_and_the_real_runner) +== MUTATION: a leaf counts as a live CA + RED on the mutant: 3 failing line(s) + FAIL: test_with_the_real_openssl_and_the_real_runner (__main__.PVECACopy.test_with_the_real_openssl_and_the_real_runner) + AssertionError: True is not false : a leaf is not a CA + FAILED (failures=1) + +the real file: 0 failing line(s) +7 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt b/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt index e90658b4..23818b57 100644 --- a/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt +++ b/documentation/audits/night-build-2026-11-14/review/redproof-hub-review-fixes.txt @@ -110,3 +110,13 @@ system_b3_test.go:63: last ask no_answer: {Text:7.0.14-28-pve · 0 rewritten · 18 h ago · last ask NO_ANSWER 1 h ago Class:warn Title:last ask: no_answer}, want amber, the failed ask AND the older record 1 mutations, 0 not a red-proof +== MUTATION: a mark is dropped the moment the box stops listing the guest (a box mints 8 alarms per report) + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.05s) + review_fixes_test.go:69: 40 alarms for five reports of invented guests, want at most 8 +== MUTATION: a mark of a guest that is gone is never dropped + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.05s) + review_fixes_test.go:82: marks for guests the box has not listed for a week: 8 (rows 8), want 0 + +2 mutations, 0 not a red-proof diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 9899634a..debfb0ac 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -8,7 +8,8 @@ (`notify.perSubjectCooldownEvents`, details `subject`): the plain `customer:type` key suppressed the second token, certificate, job, guest or cause inside the hour, and its mark was already set, so it was never mailed. (2) What a BOX sends is bounded and checked: the trust event takes only a closed mode word and a real date; one box holds at - most 8 give-up marks and a mark for a guest it no longer lists is dropped; web-certificate marks are per (household, + most 8 give-up marks, and the mark of a guest it stops listing keeps its slot for a week (so new invented guests + give 8 alarms a week, not 8 per report); web-certificate marks are per (household, name, stage), hold the end date told, are capped at 24 per household, and a name is told again only for an end date a day later. (3) `SetOSAlarmRaised` with no time DELETES the row (a cleared mark left an empty row for ever). (4) The first run of the end-date check marks what has already ended and does not mail it. (5) The off-site record: diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 4814cb34..5e68990d 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -3313,6 +3313,9 @@ const guestPowerMaxEntries = 16 // guestPowerMaxMarks bounds how many give-up marks ONE box can hold in the hub's settings at a time. const guestPowerMaxMarks = 8 +// guestPowerMarkKeep: how long the mark of a guest the box stopped listing keeps its slot. +const guestPowerMarkKeep = 7 * 24 * time.Hour + func (h *Handler) noteSignedJobOutcome(hostID, jobID, outcome, reason, op string) { switch outcome { case "rejected", "failed", "malformed": @@ -3380,20 +3383,22 @@ func (h *Handler) checkGuestPowerGaveUp(hostID, custID string, gave []hostGuestP prefix := fmt.Sprintf("guest_power:%s:", hostID) key := func(vmid int) string { return prefix + strconv.Itoa(vmid) } // The marks this box has now. The report's own guest list is the BOX's word too, so „the report lists the guest” - // bounds nothing by itself (review, 2026-11-14): the number of marks per box is capped, and a mark for a guest the - // box no longer lists is dropped — a destroyed guest must not hold a mark, or a slot, for ever. + // bounds nothing by itself (review, 2026-11-14): one box holds at most guestPowerMaxMarks marks, and a mark for a + // guest the box no longer lists keeps its slot for guestPowerMarkKeep before it is dropped. So a box that invents + // new guests at every report gets guestPowerMaxMarks alarms in that time, not that many per report — each alarm + // is mailed (the mail cooldown is per guest). A destroyed guest's mark still goes, a week later. raised := map[int]bool{} for _, k := range h.store.OSAlarmKeys(prefix) { vmid, err := strconv.Atoi(strings.TrimPrefix(k, prefix)) if err != nil { continue } - if !known[vmid] { + if !known[vmid] && time.Since(h.store.OSAlarmRaised(k)) > guestPowerMarkKeep { _ = h.store.SetOSAlarmRaised(k, time.Time{}) - h.logger.Printf("[INFO] host %s: guest %d is no longer reported — its give-up mark is dropped", hostID, vmid) + h.logger.Printf("[INFO] host %s: guest %d has not been reported for %s — its give-up mark is dropped", hostID, vmid, guestPowerMarkKeep) continue } - raised[vmid] = true + raised[vmid] = true // also a guest the box stopped listing, for guestPowerMarkKeep: its mark keeps its slot } for _, g := range gave { if g.VMID <= 0 || !known[g.VMID] || running[g.VMID] || raised[g.VMID] { diff --git a/hub/internal/api/review_fixes_test.go b/hub/internal/api/review_fixes_test.go index 256fd5da..171394b3 100644 --- a/hub/internal/api/review_fixes_test.go +++ b/hub/internal/api/review_fixes_test.go @@ -63,13 +63,23 @@ func TestGuestPowerGaveUp_ABoxCannotFillTheSettings(t *testing.T) { t.Fatalf("round %d: %d give-up marks for one box, want at most %d", round, m, guestPowerMaxMarks) } } - if n := len(r1383Events(t, st, eventGuestPowerGaveUp)); n > 5*guestPowerMaxMarks { - t.Fatalf("%d events for five reports", n) + // five reports with 80 invented guests: at most guestPowerMaxMarks ALARMS in all — not that many per report. A box + // that drops a guest from its list does not get the slot back at once. + if n := len(r1383Events(t, st, eventGuestPowerGaveUp)); n > guestPowerMaxMarks { + t.Fatalf("%d alarms for five reports of invented guests, want at most %d", n, guestPowerMaxMarks) + } + // the box now reports only its real guest: the invented marks keep their slots for a week … + h.checkGuestPowerGaveUp("h1", "c1", nil, map[int]bool{100: true}, map[int]bool{100: true}) + if marks() != guestPowerMaxMarks { + t.Fatalf("marks right after the box stopped listing the guests: %d, want %d (kept for %s)", marks(), guestPowerMaxMarks, guestPowerMarkKeep) + } + // … and are gone after it, rows and all + for _, k := range st.OSAlarmKeys("guest_power:h1:") { + _ = st.SetOSAlarmRaised(k, time.Now().Add(-guestPowerMarkKeep-time.Hour)) } - // the box now reports only its real guest: every invented mark is gone, and so is its row h.checkGuestPowerGaveUp("h1", "c1", nil, map[int]bool{100: true}, map[int]bool{100: true}) if marks() != 0 || rows() != 0 { - t.Fatalf("marks for guests the box no longer lists: %d (rows %d), want 0", marks(), rows()) + t.Fatalf("marks for guests the box has not listed for a week: %d (rows %d), want 0", marks(), rows()) } // a cleared mark leaves no row at all _ = st.SetOSAlarmRaised("probe:x", time.Now())