hub + installer (unreleased): fixes after an independent review of the night's code — per-thing mail cooldown, a box's input bounded and checked, both off-site account types on a reset, a quiet first run, strict-mode installer
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
a044eb8c16
commit
f618c591eb
31 files changed
+796
-88
No files matched your search
@@ -1003,6 +1003,18 @@ a box that is fully blind sends no host report at all, so that case still reache
|
||||
LAST, in the same write that sets `ca_file`; the installer writes a fingerprint only when the copy was not proven; and
|
||||
a config that holds both is a pinned box — the agent treats it as „not moved" and runs the move.
|
||||
|
||||
**6. What an independent review changed (2026-11-14 night, built, not released).** (a) **The copy holds one or two
|
||||
certificates.** The root step writes the new root FIRST and keeps the copy's previous first certificate behind it:
|
||||
the copy is replaced before anything can be proven with the new root, and Proxmox serves its old certificate until
|
||||
pveproxy is reloaded — without the old root, an agent that restarted in that window had no root that verified the
|
||||
served certificate and no fingerprint. Never more than two; an ended one verifies nothing. (b) **One run at a time**
|
||||
(`Ensure` is called from the start, the timer, a failed collect and a bundle install), and the root step has a
|
||||
90-second deadline — it runs on the daemon's start path and inside the report loop. (c) **One lock for every writer
|
||||
of `agent.json`** in the process (`config.FileMu`: the trust move, the PBS seed, the backup-target move): each reads
|
||||
the whole file and writes the whole file. **Left open, low:** the write is in place (not a rename), and nothing reads
|
||||
the parked copy `agent.json.pre-pve-trust` back by itself — on a full disk a failed write and a failed put-back can
|
||||
leave an empty config; the hand repair is to copy the parked file back.
|
||||
|
||||
**What changes for a pinned box when Proxmox makes a new certificate:** the first failed call names
|
||||
`pinned_cert_changed`, runs the order above, and the next call works — the box is blind for one report cycle at most,
|
||||
not until a person repairs it. `runbooks/RUNBOOK-agent-proxmox-pin-repair.md` stays the hand repair for an agent older
|
||||
|
||||
@@ -432,6 +432,10 @@ label (`audits/RETIRE-peti-2026-09-25.md` A3). `runbooks/household-leaves.md`,
|
||||
its descriptor (`enabled:false`, coordinates kept — `TestPBSDR_DisableKeepsTenancy`), and RESET and delete take it
|
||||
down by the customer id, ticked or not.
|
||||
|
||||
**After review (2026-11-14):** a tick with the OTHER type (shared, then dedicated) makes a second account, so the
|
||||
first account's record stays, and a RESET takes down every account type the config knows; the record is dropped only
|
||||
after a provider leg that really ran — a leg skipped because this hub has no provider token leaves the record.
|
||||
|
||||
**Not proven:** on a deployed hub, or against the real provider. The tests run the real save, the real RESET and the
|
||||
repository's fake provider (`hetznerapi.Fake`); 9 mutations seen red, one of them only after a missing test case was
|
||||
added (`audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt`).
|
||||
@@ -724,6 +728,11 @@ Four cases, the four the ruling names. Each leaves an event row and mails the op
|
||||
queue, so a box cannot make operator mails by clearing job ids it invents; the give-up list is read only for guests
|
||||
the same report lists (not running), 16 entries at most.
|
||||
|
||||
**The mails (after review, 2026-11-14):** each of these events tells one thing once and keeps a mark, so its mail
|
||||
cooldown is keyed per thing (`notify.perSubjectCooldownEvents`) — with the plain per-type key the second job, guest or
|
||||
skipped check inside the hour was suppressed and never told. The lost-mail scan tells only mails that failed before
|
||||
this hub process started (one that failed since still has its retry). One box holds at most 8 give-up marks.
|
||||
|
||||
**Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate
|
||||
and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red).
|
||||
|
||||
|
||||
@@ -791,7 +791,10 @@ the product warns 14 days before the two it can read itself; the rest stays a li
|
||||
What it does not do: a box with NO certificate in the store (no ACME e-mail: Traefik serves its own certificate) is not
|
||||
told — the field is absent. A box that is switched off is judged on its last report: the warning is still true (an off
|
||||
box cannot renew). A token or certificate that ended more than 14 days ago raises nothing at the first run (no news).
|
||||
**Not proven:** on a box or a deployed hub; a real failed renewal has never been seen.
|
||||
After review (2026-11-14): the FIRST run after a deploy marks what has already ended and does not mail it; the box
|
||||
reports only certificates of the household's current domain; a name is told again only for an end date at least a
|
||||
day later than the one told, and one household holds at most 24 marks — a box cannot make mails or marks by sending
|
||||
another date or new names each hour. **Not proven:** on a box or a deployed hub; a real failed renewal has never been seen.
|
||||
|
||||
### 12.2 The other sixteen `[DESIGN — proposed, not built]`
|
||||
|
||||
|
||||
@@ -734,7 +734,10 @@ so the box's last refresh is still told after an agent restart), and `last_ask`
|
||||
this agent process ended (refreshed · not_needed · failed · refused · no_answer). The agent reads one small file and
|
||||
does not call the wrapper for it; a name that is not a plain unit name is dropped. The System page's cell
|
||||
**„Security profiles”** reads „7.0.14-28-pve · 2 rewritten · 18 h ago · restarted chrony.service”; amber, and under
|
||||
„Needs attention”, when the last ask failed or got no answer. A refused ask (an older wrapper, a BYO host) is not a
|
||||
„Needs attention”, when the last ask failed or got no answer. **After review (2026-11-14):** the wrapper's record lies in a
|
||||
folder that is root-only on a box where a signed job made the folder first (read in source; to be read on a box), so
|
||||
the agent takes the last refresh from the wrapper's ANSWER to its own ask — the wrapper answers its record at every
|
||||
ask, also „not needed” — and the file only adds the detail where it can be read; `rewritten` is absent when not known. A refused ask (an older wrapper, a BYO host) is not a
|
||||
fault. No alarm is raised for a restart: it is shown, not mailed. Not seen on a box or a deployed hub.
|
||||
|
||||
**Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
== MUTATION: the trust move writes agent.json without the shared lock
|
||||
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent [build failed]
|
||||
|
||||
1 mutations, 1 not a red-proof
|
||||
== MUTATION: a give-up stays reported after onboot is switched off
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s)
|
||||
guestpower_test.go:315: onboot switched off: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}]
|
||||
== MUTATION: a give-up stays reported for a locked guest
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s)
|
||||
guestpower_test.go:315: locked by another operation: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}]
|
||||
== MUTATION: a give-up stays reported for a destroyed guest
|
||||
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/localapi [build failed]
|
||||
internal/localapi/guestpower.go:162:6: declared and not used: vmid
|
||||
|
||||
3 mutations, 1 not a red-proof
|
||||
== MUTATION: the two tokens come from two reads of the init
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestBootKey_AClockStepDoesNotMoveIt (0.00s)
|
||||
guest_boot_key_test.go:84: GuestBootTokens = "1794112550-4242", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" after 2 read(s) of the init — want both tokens from ONE read
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: the stanza knows the last refresh only from the record file (root-only on a box)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
|
||||
profilerefresh_test.go:139: stanza after an agent restart (the wrapper's answer, no detail) = {"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}
|
||||
== MUTATION: a refresh by this process is not remembered
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
|
||||
profilerefresh_test.go:123: stanza after a refresh, record file unreadable = {"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}
|
||||
== MUTATION: a readable record does not add the detail back
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
|
||||
profilerefresh_test.go:150: stanza with a readable record = {"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}
|
||||
|
||||
3 mutations, 0 not a red-proof
|
||||
== MUTATION: the trust move writes agent.json without the shared lock — done again (the first mutant left an import unused)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestAgentJSONWritersTakeTheSharedLock (0.00s)
|
||||
config_writers_lock_test.go:55: ../../internal/pvetrust/pvetrust.go writeConfig: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: a give-up stays reported for a destroyed guest — done again (the first mutant left a variable unused)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s)
|
||||
guestpower_test.go:315: destroyed: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}]
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: the copy's previous certificate is thrown away at a change (the old behaviour: no root verifies the served certificate after a restart)
|
||||
RED on the mutant: 3 failing line(s)
|
||||
FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy)
|
||||
AssertionError: Lists differ: [b'--[86 chars]----'] != [b'--[86 chars]----', b'-----BEGIN CERTIFICATE-----\nQUJDREVG[52 chars]---']
|
||||
FAILED (failures=1)
|
||||
== MUTATION: the same source is written again at every run
|
||||
RED on the mutant: 5 failing line(s)
|
||||
FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy)
|
||||
AssertionError: Tuples differ: (0, 'written', [('/etc/felhom-agent/pve-root-c[108 chars]20)]) != (0, 'same', [])
|
||||
FAIL: test_the_same_copy_is_not_written_again (__main__.PVECACopy.test_the_same_copy_is_not_written_again)
|
||||
== MUTATION: the copy grows by one certificate at every change
|
||||
RED on the mutant: 3 failing line(s)
|
||||
FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy)
|
||||
AssertionError: Lists differ: [b'--[183 chars]----', b'-----BEGIN CERTIFICATE-----\nQUJDREVG[52 chars]---'] != [b'--[183 chars]----']
|
||||
FAILED (failures=1)
|
||||
|
||||
the real file: 0 failing line(s)
|
||||
3 mutations, 0 not a red-proof
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
== MUTATION: every certificate in the store is reported, also for a name the box no longer serves
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
|
||||
web_certs_report_test.go:62: a certificate for ANOTHER domain was reported: [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu}]
|
||||
== MUTATION: the domain is matched as a plain suffix
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
|
||||
web_certs_report_test.go:65: a domain that only ENDS the same way matched: [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu}]
|
||||
|
||||
2 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,104 @@
|
||||
== MUTATION: the cooldown key has no subject (the second token of the day is never mailed)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSubjectCooldown_TwoThingsTwoMails (0.04s)
|
||||
subject_cooldown_test.go:24: mails = 1, want two (one per token) and none for the repeat: ["Customer: \nEvent: operator_token_ends_soon\nSeverity: warning\nTime: 2026-11-14 19:02 CET\nMessage: token A ends\nDetails: {\"su
|
||||
== MUTATION: every event type gets the subject suffix (other types' keys change)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSubjectCooldown_TwoThingsTwoMails (0.04s)
|
||||
subject_cooldown_test.go:37: backup_failed: its cooldown key changed (":x") — only the listed types may
|
||||
subject_cooldown_test.go:37: app_start_failed: its cooldown key changed (":x") — only the listed types may
|
||||
subject_cooldown_test.go:37: tunnel_down: its cooldown key changed (":x") — only the listed types may
|
||||
|
||||
2 mutations, 0 not a red-proof
|
||||
== MUTATION: the box's mode word reaches the event unchecked
|
||||
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/api [build failed]
|
||||
== MUTATION: the box's date text reaches the event unchecked
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_OnlyClosedWordsReachTheEvent (0.08s)
|
||||
review_fixes_test.go:25: the box's own text "send money" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: other). The root CA copy ends in 3 day(s) (send money to X). Design: `03` §12a.{"ca_not_afte
|
||||
review_fixes_test.go:29: unknown words must read as other / empty: Trust in Proxmox on h1: a cause this hub does not know (mode: other). The root CA copy ends in 3 day(s) (send money to X). Design: `03` §12a.{"ca_not_after":"send
|
||||
== MUTATION: no cap on a box's give-up marks
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.05s)
|
||||
review_fixes_test.go:63: round 0: 16 give-up marks for one box, want at most 8
|
||||
== MUTATION: a mark for a guest the box no longer lists stays for ever
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.04s)
|
||||
review_fixes_test.go:72: marks for guests the box no longer lists: 8 (rows 8), want 0
|
||||
|
||||
4 mutations, 1 not a red-proof
|
||||
== MUTATION: a cleared alarm leaves an empty row
|
||||
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
|
||||
== MUTATION: the key list matches with LIKE-style prefix of another id (no exact prefix)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s)
|
||||
d73_gone_rows_test.go:168: web-certificate marks of cust-a: 2, want 0
|
||||
d73_gone_rows_test.go:168: web-certificate marks of cust-b: 2, want 1
|
||||
|
||||
2 mutations, 1 not a red-proof
|
||||
== MUTATION: a household's web-certificate marks outlive it
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s)
|
||||
d73_gone_rows_test.go:168: web-certificate marks of cust-a: 1, want 0
|
||||
== MUTATION: the purge also removes the marks of a household whose id starts the same way
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s)
|
||||
d73_gone_rows_test.go:168: web-certificate marks of cust-a2: 0, want 1
|
||||
|
||||
2 mutations, 0 not a red-proof
|
||||
== MUTATION: a first run mails ENDED for a token that ended before the deploy
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s)
|
||||
end_dates_test.go:173: the first run: [|operator_token_ends_soon|error|The operator access token „replaced” ENDED on 2026-11-11 and is not revoked: every script that still uses it gets 401. Do: if it was replaced, revoke it on the
|
||||
== MUTATION: a box gets a new mail for every new end date
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s)
|
||||
end_dates_test.go:195: a box that moves its end date by a minute each hour got 48 mails in two days (want at most a few: a new mail needs a date a day later)
|
||||
== MUTATION: no cap on a household's web-certificate marks
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s)
|
||||
end_dates_test.go:208: 192 invented names made 192 marks, want at most 24 for one household
|
||||
|
||||
3 mutations, 0 not a red-proof
|
||||
== MUTATION: a tick with the other type drops the first account's record
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestKeepAccountRecord_FourCases (0.00s)
|
||||
account_record_test.go:47: unticked shared, then ticked as DEDICATED: the shared account's record stays: record <nil>, want user u1-sub3 unticked_at 2026-11-01T00:00:00Z ({"offsite":{"enabled":true,"type":"dedicated","user":"u9","
|
||||
account_record_test.go:47: switched from shared to dedicated in ONE save: the shared account gets a record: record <nil>, want user u1-sub3 unticked_at 2026-11-14T20:00:00Z ({"offsite":{"enabled":true,"type":"dedicated","user":"u9
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: the record is dropped also when the provider leg was skipped
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo (0.05s)
|
||||
d71_offsite_account_record_test.go:193: the provider leg was SKIPPED and the account's record is gone: {}
|
||||
== MUTATION: the RESET takes down only the first account type
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo (0.05s)
|
||||
d71_offsite_account_record_test.go:197: a household with a dedicated tier AND a kept shared record: the RESET would take down [dedicated], want both
|
||||
== MUTATION: the record is never dropped after a real takedown
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.06s)
|
||||
d71_offsite_account_record_test.go:131: the record outlived the account: {"logging":{"level":"debug"},"offsite_account":{"type":"shared","user":"u629193-sub1000","host":"u629193-sub1000.your-storagebox.de","unticked_at":"2026-11-1
|
||||
|
||||
3 mutations, 0 not a red-proof
|
||||
== MUTATION: a mail that failed in this process is told as lost
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_AMailThatFailedInThisProcessIsNotLost (0.04s)
|
||||
lost_operator_mail_test.go:109: a mail that failed AFTER this process started was told as lost (1)
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: a cleared alarm leaves an empty row — done again: the first run was GREEN (the test read through a function that hides empty rows; a test that counts the rows was added)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestOSAlarm_AClearedMarkLeavesNoRow (0.04s)
|
||||
d73_gone_rows_test.go:253: a cleared mark left 1 row(s)
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: the box's mode word reaches the event unchecked — done again (the first mutant did not compile)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_OnlyClosedWordsReachTheEvent (0.08s)
|
||||
review_fixes_test.go:25: the box's own text "MODE" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: <b>MODE</b> call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":""
|
||||
review_fixes_test.go:25: the box's own text "<b>" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: <b>MODE</b> call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"",
|
||||
review_fixes_test.go:29: unknown words must read as other / empty: Trust in Proxmox on h1: a cause this hub does not know (mode: <b>MODE</b> call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"","c
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,9 @@
|
||||
== MUTATION: a failed fingerprint read ends the installer (the old line)
|
||||
RED on the mutant: 1 failing line(s)
|
||||
FAIL test_pve_trust_no_answer_from_proxmox_does_not_end_the_installer --
|
||||
== MUTATION: the proof is a pipe into grep -q again (a good copy can read as not verified under pipefail)
|
||||
*** NOT A RED-PROOF ***: 0 failing line(s)
|
||||
|
||||
the real file: 0 failing line(s)
|
||||
2 mutations, 1 not a red-proof
|
||||
NOTE: the second mutation is NOT a red-proof and is left here on purpose. The defect it models (grep -q closes the pipe while s_client still writes; pipefail then fails the pipeline for a good copy) was reasoned by the reviewer and could not be reproduced in this test in 2 runs. The fix (read the answer whole, search after) removes the pipe, so the question does not arise; it is NOT proven that the old line ever failed.
|
||||
Reference in new issue
Block a user