hub + installer (unreleased): fixes after an independent review of the night's code — per-thing mail cooldown, a box's input bounded and checked, both off-site account types on a reset, a quiet first run, strict-mode installer

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-14 19:08:55 +01:00
1 parent a044eb8c16
commit f618c591eb
31 files changed
+796 -88

No files matched your search

@@ -1003,6 +1003,18 @@ a box that is fully blind sends no host report at all, so that case still reache
LAST, in the same write that sets `ca_file`; the installer writes a fingerprint only when the copy was not proven; and
a config that holds both is a pinned box — the agent treats it as „not moved" and runs the move.
**6. What an independent review changed (2026-11-14 night, built, not released).** (a) **The copy holds one or two
certificates.** The root step writes the new root FIRST and keeps the copy's previous first certificate behind it:
the copy is replaced before anything can be proven with the new root, and Proxmox serves its old certificate until
pveproxy is reloaded — without the old root, an agent that restarted in that window had no root that verified the
served certificate and no fingerprint. Never more than two; an ended one verifies nothing. (b) **One run at a time**
(`Ensure` is called from the start, the timer, a failed collect and a bundle install), and the root step has a
90-second deadline — it runs on the daemon's start path and inside the report loop. (c) **One lock for every writer
of `agent.json`** in the process (`config.FileMu`: the trust move, the PBS seed, the backup-target move): each reads
the whole file and writes the whole file. **Left open, low:** the write is in place (not a rename), and nothing reads
the parked copy `agent.json.pre-pve-trust` back by itself — on a full disk a failed write and a failed put-back can
leave an empty config; the hand repair is to copy the parked file back.
**What changes for a pinned box when Proxmox makes a new certificate:** the first failed call names
`pinned_cert_changed`, runs the order above, and the next call works — the box is blind for one report cycle at most,
not until a person repairs it. `runbooks/RUNBOOK-agent-proxmox-pin-repair.md` stays the hand repair for an agent older
@@ -432,6 +432,10 @@ label (`audits/RETIRE-peti-2026-09-25.md` A3). `runbooks/household-leaves.md`,
its descriptor (`enabled:false`, coordinates kept — `TestPBSDR_DisableKeepsTenancy`), and RESET and delete take it
down by the customer id, ticked or not.
**After review (2026-11-14):** a tick with the OTHER type (shared, then dedicated) makes a second account, so the
first account's record stays, and a RESET takes down every account type the config knows; the record is dropped only
after a provider leg that really ran — a leg skipped because this hub has no provider token leaves the record.
**Not proven:** on a deployed hub, or against the real provider. The tests run the real save, the real RESET and the
repository's fake provider (`hetznerapi.Fake`); 9 mutations seen red, one of them only after a missing test case was
added (`audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt`).
@@ -724,6 +728,11 @@ Four cases, the four the ruling names. Each leaves an event row and mails the op
queue, so a box cannot make operator mails by clearing job ids it invents; the give-up list is read only for guests
the same report lists (not running), 16 entries at most.
**The mails (after review, 2026-11-14):** each of these events tells one thing once and keeps a mark, so its mail
cooldown is keyed per thing (`notify.perSubjectCooldownEvents`) — with the plain per-type key the second job, guest or
skipped check inside the hour was suppressed and never told. The lost-mail scan tells only mails that failed before
this hub process started (one that failed since still has its retry). One box holds at most 8 give-up marks.
**Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate
and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red).
@@ -791,7 +791,10 @@ the product warns 14 days before the two it can read itself; the rest stays a li
What it does not do: a box with NO certificate in the store (no ACME e-mail: Traefik serves its own certificate) is not
told — the field is absent. A box that is switched off is judged on its last report: the warning is still true (an off
box cannot renew). A token or certificate that ended more than 14 days ago raises nothing at the first run (no news).
**Not proven:** on a box or a deployed hub; a real failed renewal has never been seen.
After review (2026-11-14): the FIRST run after a deploy marks what has already ended and does not mail it; the box
reports only certificates of the household's current domain; a name is told again only for an end date at least a
day later than the one told, and one household holds at most 24 marks — a box cannot make mails or marks by sending
another date or new names each hour. **Not proven:** on a box or a deployed hub; a real failed renewal has never been seen.
### 12.2 The other sixteen `[DESIGN — proposed, not built]`
+4 -1
View File
@@ -734,7 +734,10 @@ so the box's last refresh is still told after an agent restart), and `last_ask`
this agent process ended (refreshed · not_needed · failed · refused · no_answer). The agent reads one small file and
does not call the wrapper for it; a name that is not a plain unit name is dropped. The System page's cell
**„Security profiles”** reads „7.0.14-28-pve · 2 rewritten · 18 h ago · restarted chrony.service”; amber, and under
„Needs attention”, when the last ask failed or got no answer. A refused ask (an older wrapper, a BYO host) is not a
„Needs attention”, when the last ask failed or got no answer. **After review (2026-11-14):** the wrapper's record lies in a
folder that is root-only on a box where a signed job made the folder first (read in source; to be read on a box), so
the agent takes the last refresh from the wrapper's ANSWER to its own ask — the wrapper answers its record at every
ask, also „not needed” — and the file only adds the detail where it can be read; `rewritten` is absent when not known. A refused ask (an older wrapper, a BYO host) is not a
fault. No alarm is raised for a restart: it is shown, not mailed. Not seen on a box or a deployed hub.
**Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced
@@ -0,0 +1,69 @@
== MUTATION: the trust move writes agent.json without the shared lock
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent [build failed]
1 mutations, 1 not a red-proof
== MUTATION: a give-up stays reported after onboot is switched off
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s)
guestpower_test.go:315: onboot switched off: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}]
== MUTATION: a give-up stays reported for a locked guest
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s)
guestpower_test.go:315: locked by another operation: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}]
== MUTATION: a give-up stays reported for a destroyed guest
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/localapi [build failed]
internal/localapi/guestpower.go:162:6: declared and not used: vmid
3 mutations, 1 not a red-proof
== MUTATION: the two tokens come from two reads of the init
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestBootKey_AClockStepDoesNotMoveIt (0.00s)
guest_boot_key_test.go:84: GuestBootTokens = "1794112550-4242", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" after 2 read(s) of the init — want both tokens from ONE read
1 mutations, 0 not a red-proof
== MUTATION: the stanza knows the last refresh only from the record file (root-only on a box)
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:139: stanza after an agent restart (the wrapper's answer, no detail) = {"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}
== MUTATION: a refresh by this process is not remembered
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:123: stanza after a refresh, record file unreadable = {"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}
== MUTATION: a readable record does not add the detail back
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:150: stanza with a readable record = {"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}
3 mutations, 0 not a red-proof
== MUTATION: the trust move writes agent.json without the shared lock — done again (the first mutant left an import unused)
RED on the mutant, GREEN on the real file
--- FAIL: TestAgentJSONWritersTakeTheSharedLock (0.00s)
config_writers_lock_test.go:55: ../../internal/pvetrust/pvetrust.go writeConfig: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()
1 mutations, 0 not a red-proof
== MUTATION: a give-up stays reported for a destroyed guest — done again (the first mutant left a variable unused)
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs (0.00s)
guestpower_test.go:315: destroyed: the give-up is still reported: [{VMID:9201 Since:2026-11-14T02:30:00Z Attempts:3}]
1 mutations, 0 not a red-proof
== MUTATION: the copy's previous certificate is thrown away at a change (the old behaviour: no root verifies the served certificate after a restart)
RED on the mutant: 3 failing line(s)
FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy)
AssertionError: Lists differ: [b'--[86 chars]----'] != [b'--[86 chars]----', b'-----BEGIN CERTIFICATE-----\nQUJDREVG[52 chars]---']
FAILED (failures=1)
== MUTATION: the same source is written again at every run
RED on the mutant: 5 failing line(s)
FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy)
AssertionError: Tuples differ: (0, 'written', [('/etc/felhom-agent/pve-root-c[108 chars]20)]) != (0, 'same', [])
FAIL: test_the_same_copy_is_not_written_again (__main__.PVECACopy.test_the_same_copy_is_not_written_again)
== MUTATION: the copy grows by one certificate at every change
RED on the mutant: 3 failing line(s)
FAIL: test_a_renewed_ca_refreshes_the_copy (__main__.PVECACopy.test_a_renewed_ca_refreshes_the_copy)
AssertionError: Lists differ: [b'--[183 chars]----', b'-----BEGIN CERTIFICATE-----\nQUJDREVG[52 chars]---'] != [b'--[183 chars]----']
FAILED (failures=1)
the real file: 0 failing line(s)
3 mutations, 0 not a red-proof
@@ -0,0 +1,10 @@
== MUTATION: every certificate in the store is reported, also for a name the box no longer serves
RED on the mutant, GREEN on the real file
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
web_certs_report_test.go:62: a certificate for ANOTHER domain was reported: [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu}]
== MUTATION: the domain is matched as a plain suffix
RED on the mutant, GREEN on the real file
--- FAIL: TestWebCertsForReport_PathFieldAndBrokenStore (0.00s)
web_certs_report_test.go:65: a domain that only ENDS the same way matched: [{Domain:*.example.hu NotAfter:2027-01-19T01:46:00Z Issuer:*.example.hu}]
2 mutations, 0 not a red-proof
@@ -0,0 +1,104 @@
== MUTATION: the cooldown key has no subject (the second token of the day is never mailed)
RED on the mutant, GREEN on the real file
--- FAIL: TestSubjectCooldown_TwoThingsTwoMails (0.04s)
subject_cooldown_test.go:24: mails = 1, want two (one per token) and none for the repeat: ["Customer: \nEvent: operator_token_ends_soon\nSeverity: warning\nTime: 2026-11-14 19:02 CET\nMessage: token A ends\nDetails: {\"su
== MUTATION: every event type gets the subject suffix (other types' keys change)
RED on the mutant, GREEN on the real file
--- FAIL: TestSubjectCooldown_TwoThingsTwoMails (0.04s)
subject_cooldown_test.go:37: backup_failed: its cooldown key changed (":x") — only the listed types may
subject_cooldown_test.go:37: app_start_failed: its cooldown key changed (":x") — only the listed types may
subject_cooldown_test.go:37: tunnel_down: its cooldown key changed (":x") — only the listed types may
2 mutations, 0 not a red-proof
== MUTATION: the box's mode word reaches the event unchecked
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/api [build failed]
== MUTATION: the box's date text reaches the event unchecked
RED on the mutant, GREEN on the real file
--- FAIL: TestProxmoxTrust_OnlyClosedWordsReachTheEvent (0.08s)
review_fixes_test.go:25: the box's own text "send money" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: other). The root CA copy ends in 3 day(s) (send money to X). Design: `03` §12a.{"ca_not_afte
review_fixes_test.go:29: unknown words must read as other / empty: Trust in Proxmox on h1: a cause this hub does not know (mode: other). The root CA copy ends in 3 day(s) (send money to X). Design: `03` §12a.{"ca_not_after":"send
== MUTATION: no cap on a box's give-up marks
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.05s)
review_fixes_test.go:63: round 0: 16 give-up marks for one box, want at most 8
== MUTATION: a mark for a guest the box no longer lists stays for ever
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPowerGaveUp_ABoxCannotFillTheSettings (0.04s)
review_fixes_test.go:72: marks for guests the box no longer lists: 8 (rows 8), want 0
4 mutations, 1 not a red-proof
== MUTATION: a cleared alarm leaves an empty row
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
== MUTATION: the key list matches with LIKE-style prefix of another id (no exact prefix)
RED on the mutant, GREEN on the real file
--- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s)
d73_gone_rows_test.go:168: web-certificate marks of cust-a: 2, want 0
d73_gone_rows_test.go:168: web-certificate marks of cust-b: 2, want 1
2 mutations, 1 not a red-proof
== MUTATION: a household's web-certificate marks outlive it
RED on the mutant, GREEN on the real file
--- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s)
d73_gone_rows_test.go:168: web-certificate marks of cust-a: 1, want 0
== MUTATION: the purge also removes the marks of a household whose id starts the same way
RED on the mutant, GREEN on the real file
--- FAIL: TestD73_PurgeRemovesTheHouseholdsRows_JournalStays (0.05s)
d73_gone_rows_test.go:168: web-certificate marks of cust-a2: 0, want 1
2 mutations, 0 not a red-proof
== MUTATION: a first run mails ENDED for a token that ended before the deploy
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s)
end_dates_test.go:173: the first run: [|operator_token_ends_soon|error|The operator access token „replaced” ENDED on 2026-11-11 and is not revoked: every script that still uses it gets 401. Do: if it was replaced, revoke it on the
== MUTATION: a box gets a new mail for every new end date
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s)
end_dates_test.go:195: a box that moves its end date by a minute each hour got 48 mails in two days (want at most a few: a new mail needs a date a day later)
== MUTATION: no cap on a household's web-certificate marks
RED on the mutant, GREEN on the real file
--- FAIL: TestEndDates_FirstRunAndABoxCannotMintMarks (0.00s)
end_dates_test.go:208: 192 invented names made 192 marks, want at most 24 for one household
3 mutations, 0 not a red-proof
== MUTATION: a tick with the other type drops the first account's record
RED on the mutant, GREEN on the real file
--- FAIL: TestKeepAccountRecord_FourCases (0.00s)
account_record_test.go:47: unticked shared, then ticked as DEDICATED: the shared account's record stays: record <nil>, want user u1-sub3 unticked_at 2026-11-01T00:00:00Z ({"offsite":{"enabled":true,"type":"dedicated","user":"u9","
account_record_test.go:47: switched from shared to dedicated in ONE save: the shared account gets a record: record <nil>, want user u1-sub3 unticked_at 2026-11-14T20:00:00Z ({"offsite":{"enabled":true,"type":"dedicated","user":"u9
1 mutations, 0 not a red-proof
== MUTATION: the record is dropped also when the provider leg was skipped
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo (0.05s)
d71_offsite_account_record_test.go:193: the provider leg was SKIPPED and the account's record is gone: {}
== MUTATION: the RESET takes down only the first account type
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_RecordSurvivesASkippedLeg_AndBothTypesGo (0.05s)
d71_offsite_account_record_test.go:197: a household with a dedicated tier AND a kept shared record: the RESET would take down [dedicated], want both
== MUTATION: the record is never dropped after a real takedown
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.06s)
d71_offsite_account_record_test.go:131: the record outlived the account: {"logging":{"level":"debug"},"offsite_account":{"type":"shared","user":"u629193-sub1000","host":"u629193-sub1000.your-storagebox.de","unticked_at":"2026-11-1
3 mutations, 0 not a red-proof
== MUTATION: a mail that failed in this process is told as lost
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_AMailThatFailedInThisProcessIsNotLost (0.04s)
lost_operator_mail_test.go:109: a mail that failed AFTER this process started was told as lost (1)
1 mutations, 0 not a red-proof
== MUTATION: a cleared alarm leaves an empty row — done again: the first run was GREEN (the test read through a function that hides empty rows; a test that counts the rows was added)
RED on the mutant, GREEN on the real file
--- FAIL: TestOSAlarm_AClearedMarkLeavesNoRow (0.04s)
d73_gone_rows_test.go:253: a cleared mark left 1 row(s)
1 mutations, 0 not a red-proof
== MUTATION: the box's mode word reaches the event unchecked — done again (the first mutant did not compile)
RED on the mutant, GREEN on the real file
--- FAIL: TestProxmoxTrust_OnlyClosedWordsReachTheEvent (0.08s)
review_fixes_test.go:25: the box's own text "MODE" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: <b>MODE</b> call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":""
review_fixes_test.go:25: the box's own text "<b>" reached the event: Trust in Proxmox on h1: a cause this hub does not know (mode: <b>MODE</b> call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"",
review_fixes_test.go:29: unknown words must read as other / empty: Trust in Proxmox on h1: a cause this hub does not know (mode: <b>MODE</b> call +36…). The root CA copy ends in 3 day(s) (). Design: `03` §12a.{"ca_not_after":"","c
1 mutations, 0 not a red-proof
@@ -0,0 +1,9 @@
== MUTATION: a failed fingerprint read ends the installer (the old line)
RED on the mutant: 1 failing line(s)
FAIL test_pve_trust_no_answer_from_proxmox_does_not_end_the_installer --
== MUTATION: the proof is a pipe into grep -q again (a good copy can read as not verified under pipefail)
*** NOT A RED-PROOF ***: 0 failing line(s)
the real file: 0 failing line(s)
2 mutations, 1 not a red-proof
NOTE: the second mutation is NOT a red-proof and is left here on purpose. The defect it models (grep -q closes the pipe while s_client still writes; pipefail then fails the pipeline for a good copy) was reasoned by the reviewer and could not be reproduced in this test in 2 runs. The fix (read the answer whole, search after) removes the pipe, so the question does not arise; it is NOT proven that the old line ever failed.