hub (unreleased): four silent failures tell the operator (R-1383, D78)
gates / gates (push) Successful in 2m9s
gates / gates (push) Successful in 2m9s
A hub daily check skipped while the hub was down; an operator mail that was never sent; a signed job a box did not take; a signed job a box refused, and the guest-power watchdog's give-up (both from the agent). Operator-only. No new table. Not deployed. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
88e636445a
commit
1886030782
22 files changed
+1269
-2
No files matched your search
@@ -645,6 +645,24 @@ never „now + 24 h", and the slot that ran last never runs again. On a 25-hour
|
||||
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
||||
2026-10-25 and 2027-03-28.
|
||||
|
||||
### 16.6 A job that does not run, gives up, or loses its state at a restart tells the operator [FACT — built, not released: hub `main` after v0.181.0 and agent `main` after 0.174.0, 2026-11-14; D78, `09` §3 decision 299, R-1383]
|
||||
|
||||
Four cases, the four the ruling names. Each leaves an event row and mails the operator only
|
||||
(`notify.operatorOnlyEvents`). None needs a new table: the state is the hub's time-by-key settings
|
||||
(`OSAlarmRaised`), the notification log and the signed-job queue.
|
||||
|
||||
| Case | Who sees it | How | Event | What it does NOT do |
|
||||
|---|---|---|---|---|
|
||||
| A hub daily check's slot passed while the hub was down | the hub, at its next start | `monitor.DailyLedger` keeps „the slot that last ran to its end” per watched job (`deadline-check`, `offsite-key-audit`) across a restart; `scheduleDaily` asks it at its start and tells it after each run | `hub_daily_job_skipped` (warning), once per missed slot | it does not run the missed check late. The snapshot and the prune are not watched (they make up for a missed slot by themselves). A job it has never seen is watched from the first start, not judged |
|
||||
| An operator mail was never sent | the hub, at its next start | `Dispatcher.ReportLostOperatorMails`: at a start no retry timer is alive, so every operator mail of the last 48 hours whose newest delivery record is `failed` was never sent. ONE mail carries each lost mail's own sentence; the scan's time is kept, so a second restart does not repeat it | `operator_mail_lost` (error) | a mail that gave up while the hub keeps running leaves an event row at once and is MAILED only at the next start. The summary is a mail like any other and can be lost too; the next start lists it. A household mail is still never retried (R-1361) |
|
||||
| A signed job a box never took | the hub, on the 60 s sweep | `monitor.SignedJobWaitChecker`: a job still in the queue two hours after it was put there (a box asks every 15 minutes at the latest). The hub does not open a signed job, so it judges by age, not by the job's own end time | `signed_job_not_taken` (warning), once per job | it does not remove the job |
|
||||
| A signed job a box took and refused, or whose run failed | the box's agent | the agent's clearing call (`DELETE …/jobs/<id>`) now carries `outcome` (done · rejected · failed · malformed), a `reason` from a closed list (`signedjobs.ReasonCodes`) and the op's name in its query; before, „ran” and „refused” looked the same | `signed_job_refused` (error), once per job. The two clock reasons name the box's clock | an agent older than release 3 says nothing, and nothing is raised. The box's own words never reach a mail (unknown reason → „other”) |
|
||||
| The guest-power watchdog gave up | the box's agent | the host report's `guest_power_gave_up` list (vmid, since, attempts); the watchdog's record is memory | `guest_power_gave_up` (error), once per down period; `guest_power_ok` (info) when a later report shows that guest running | „gone from the list” alone never clears (an agent restart empties the list while the guest is still down) |
|
||||
|
||||
**Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate
|
||||
and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red).
|
||||
|
||||
|
||||
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
|
||||
|
||||
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
|
||||
|
||||
@@ -374,6 +374,11 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
||||
| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
|
||||
| `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` |
|
||||
| `box_clock_wrong` | warning | **[built, not deployed — hub `main` after v0.181.0; needs an agent that stamps `sent_at`, built and not released]** the box's clock at the moment of sending is more than **one minute** from the hub's clock at arrival, in either direction (D79, `09` §3 decision 300); told once per direction. The hub cannot tell whose clock is wrong: a second box off in the same direction makes the alarm say „check the hub's own clock first”. An agent without `sent_at` is never judged | the first report within **30 seconds** → `box_clock_ok` (info) | `api/box_clock_test.go`; agent `hub/client_sentat_test.go` |
|
||||
| `hub_daily_job_skipped` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: the newest slot of `deadline-check` or `offsite-key-audit` passed while the hub was not running; once per missed slot | none (the next slot runs) | `monitor/daily_ledger_test.go`, `cmd/hub/r1383_wiring_test.go` |
|
||||
| `operator_mail_lost` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: operator mails of the last 48 h whose newest delivery record is `failed` — one mail for all, with each lost mail's own sentence; and a row (no mail) when three retries gave up | none | `notify/lost_operator_mail_test.go` |
|
||||
| `signed_job_not_taken` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** a signed job is still in a box's queue after two hours; once per job | none (the job leaves the queue when the box takes it) | `monitor/signed_job_waits_test.go` |
|
||||
| `signed_job_refused` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** needs an agent of release 3: the box cleared a signed job and said it was rejected, malformed, or its run failed; the cause is one of a closed list; once per job | none | `api/r1383_box_told_test.go`; agent `signedjobs/outcome_test.go` |
|
||||
| `guest_power_gave_up` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** needs an agent of release 3: the host report lists a guest the guest-power watchdog stopped trying to start; once per down period | a later report shows that guest running → `guest_power_ok` (info) | `api/r1383_box_told_test.go`; agent `localapi/guestpower_test.go` |
|
||||
| `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` |
|
||||
| `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` |
|
||||
| `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` |
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
== MUTATION: the give-up is not listed (the journal line stays the only trace)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
|
||||
guestpower_test.go:271: give-up list = [], want guest 9201, 3 attempts, since 2026-11-14T02:30:00Z (the give-up's time, not a later tick's)
|
||||
== MUTATION: a guest the watchdog still tries is listed as given up
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
|
||||
guestpower_test.go:261: after 1 failed start(s) the watchdog still tries, but it reports a give-up: [{VMID:9201 Since:0001-01-01T00:00:00Z Attempts:1}]
|
||||
== MUTATION: the give-up's time is not recorded
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
|
||||
guestpower_test.go:271: give-up list = [{VMID:9201 Since:0001-01-01T00:00:00Z Attempts:3}], want guest 9201, 3 attempts, since 2026-11-14T02:30:00Z (the give-up's time, not a later tick's)
|
||||
== MUTATION: an empty list is sent as a typed nil (the report would carry null)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
|
||||
guestpower_test.go:256: before any attempt the stanza is []localapi.GuestPowerGiveUp(nil), want an untyped nil (the report omits the field)
|
||||
|
||||
4 mutations, 0 not a red-proof
|
||||
== MUTATION: the outcome is not sent
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestCompleteJob_OutcomeRidesTheQuery (0.00s)
|
||||
client_sentat_test.go:74: requests = ["/api/v1/hosts/demo-host-01/jobs/j1?" "/api/v1/hosts/demo-host-01/jobs/j2?"], want ["/api/v1/hosts/demo-host-01/jobs/j1?" "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&r
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
== MUTATION: main does not wire the give-up into the report
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestMainWiresPVETrust (0.01s)
|
||||
pvetrust_wiring_test.go:64: main.go wires "guest-power give-up → report" 0 time(s), want at least 1 (the two-call ones: at start and after a bundle install)
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,34 @@
|
||||
== MUTATION: a refused job is cleared with no word (the old behaviour)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
|
||||
outcome_test.go:70: job "forged": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:malformed Op:storage_wipe}
|
||||
outcome_test.go:70: job "expired": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:expired Op:storage_wipe}
|
||||
outcome_test.go:70: job "early": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
|
||||
== MUTATION: a refused job is told as done
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
|
||||
outcome_test.go:70: job "early": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
|
||||
outcome_test.go:70: job "other-box": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:target Op:storage_wipe}
|
||||
outcome_test.go:70: job "forged": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:malformed Op:storage_wipe}
|
||||
== MUTATION: the runner never uses the outcome call
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
|
||||
outcome_test.go:70: job "early": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
|
||||
outcome_test.go:70: job "other-box": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:target Op:storage_wipe}
|
||||
outcome_test.go:70: job "forged": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:malformed Op:storage_wipe}
|
||||
== MUTATION: the two clock reasons are swapped
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
|
||||
outcome_test.go:70: job "expired": the hub was told {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} (told=true), want {Outcome:rejected Reason:expired Op:storage_wipe}
|
||||
outcome_test.go:70: job "early": the hub was told {Outcome:rejected Reason:expired Op:storage_wipe} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
|
||||
== MUTATION: a failed run is told as done
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
|
||||
outcome_test.go:70: job "run-fails": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:failed Reason:execution_failed Op:storage_wipe}
|
||||
== MUTATION: an unknown refusal sends the error's own text
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestReasonCode_ClosedList (0.00s)
|
||||
outcome_test.go:92: reasonCode({Allowed:false Disposition: Reason:rejected Verified:<nil> Err:something new}) = "something new", not on ReasonCodes
|
||||
outcome_test.go:96: an unknown refusal must be told as „other”, never as the error's own text
|
||||
|
||||
6 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,40 @@
|
||||
== MUTATION: a refused job is not told (executed or rejected look the same)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
|
||||
r1383_box_told_test.go:52: events=0, want ONE for the refused job
|
||||
== MUTATION: a job that ran is told as refused
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.05s)
|
||||
r1383_box_told_test.go:43: a job that ran, or an agent that says nothing, raised 2 event(s): [{ID:1 CustomerID:c1 EventType:signed_job_refused Severity:error Message:The signed job ran (agent_update) for h1 was REFUSED: the box ga
|
||||
== MUTATION: a repeated DELETE is told twice
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
|
||||
r1383_box_told_test.go:52: events=2, want ONE for the refused job
|
||||
== MUTATION: the box's own reason text reaches the event
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
|
||||
r1383_box_told_test.go:74: the failed run or the unknown words are not told as designed:
|
||||
r1383_box_told_test.go:77: the box's own text reached an event:
|
||||
== MUTATION: the op name is not checked
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
|
||||
r1383_box_told_test.go:74: the failed run or the unknown words are not told as designed:
|
||||
r1383_box_told_test.go:77: the box's own text reached an event:
|
||||
== MUTATION: the give-up is told at every report
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s)
|
||||
r1383_box_told_test.go:113: two reports of the same give-up: gave_up=2 ok=0, want ONE alarm
|
||||
== MUTATION: the give-up is never told
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.04s)
|
||||
r1383_box_told_test.go:113: two reports of the same give-up: gave_up=0 ok=0, want ONE alarm
|
||||
== MUTATION: the entry vanishing clears the alarm (an agent restart reads as a recovery)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s)
|
||||
r1383_box_told_test.go:121: the entry vanished while the guest is still stopped: gave_up=1 ok=1 — that is not a recovery
|
||||
== MUTATION: the alarm never clears
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s)
|
||||
r1383_box_told_test.go:130: after the guest runs again: gave_up=1 ok=0, want one recovery
|
||||
|
||||
9 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,48 @@
|
||||
== MUTATION: the ledger never tells (the old behaviour: last run is forgotten at a restart)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
|
||||
daily_ledger_test.go:59: events=0, want ONE for the slot the hub was down at: []
|
||||
== MUTATION: a slot that ran is told as skipped
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
|
||||
daily_ledger_test.go:54: a slot that ran was told as skipped: [|hub_daily_job_skipped|warning|hub|The hub's daily check „deadline-check” did NOT run at 2026-11-10 05:00 CET: the hub was not running at that minute. It last ran at 2
|
||||
== MUTATION: the same missed slot is told at every restart
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
|
||||
daily_ledger_test.go:73: the same missed slot was told 2 times
|
||||
== MUTATION: the run is not recorded
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
|
||||
daily_ledger_test.go:54: a slot that ran was told as skipped: [|hub_daily_job_skipped|warning|hub|The hub's daily check „deadline-check” did NOT run at 2026-11-10 05:00 CET: the hub was not running at that minute. It last ran at 2
|
||||
== MUTATION: every job is watched
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestDailyLedger_UnwatchedJobAndNilLedger (0.00s)
|
||||
daily_ledger_test.go:95: an unwatched job: events=[] state=map[daily_last:prune:2026-11-09 03:00:00 +0100 CET]
|
||||
== MUTATION: the slot is taken as now minus 24 h, not a wall-clock time
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestPrevDailySlot_HourChangeDays (0.00s)
|
||||
daily_ledger_test.go:113: PrevDailySlot(2026-10-25T04:30:00+01:00) = 2026-10-24T06:00:00+02:00, want 2026-10-24T05:00:00+02:00
|
||||
|
||||
6 mutations, 0 not a red-proof
|
||||
== MUTATION: the scheduler does not ask the ledger at its start
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
|
||||
r1383_wiring_test.go:74: scheduleDaily calls the ledger: Started 0, Ran 1 — want 1 and 1
|
||||
== MUTATION: the scheduler does not record a run
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
|
||||
r1383_wiring_test.go:74: scheduleDaily calls the ledger: Started 1, Ran 0 — want 1 and 1
|
||||
== MUTATION: the lost-mail scan is never started
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
|
||||
r1383_wiring_test.go:70: main.go: lost-mail scan — found 0 time(s), want 1
|
||||
== MUTATION: the signed-job check is not on the sweep
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
|
||||
r1383_wiring_test.go:70: main.go: signed-job sweep — found 0 time(s), want 1
|
||||
== MUTATION: the ledger watches a job name nobody runs
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
|
||||
r1383_wiring_test.go:70: main.go: watched "offsite-key-audit" — found 0 time(s), want 1
|
||||
|
||||
5 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,38 @@
|
||||
== MUTATION: the scan lists nothing
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
|
||||
lost_operator_mail_test.go:70: lost mails told = 0, want 1 (the one whose retries the restart ate; not the one that went out on a retry)
|
||||
== MUTATION: the scan's time is not kept — a second start tells the same mail
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
|
||||
lost_operator_mail_test.go:89: a second start told 1 again (mails now 1)
|
||||
== MUTATION: the summary is a row only, no mail
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
|
||||
lost_operator_mail_test.go:73: mails sent at the start = [], want ONE to the operator
|
||||
== MUTATION: the summary leaves no row
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
|
||||
lost_operator_mail_test.go:85: event rows = [], want one operator_mail_lost row
|
||||
|
||||
4 mutations, 0 not a red-proof
|
||||
== MUTATION: a mail that went out on a retry counts as lost (every failed row)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.05s)
|
||||
lost_operator_mail_test.go:70: lost mails told = 2, want 1 (the one whose retries the restart ate; not the one that went out on a retry)
|
||||
== MUTATION: a cooldown's suppressed row hides the failed mail
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
|
||||
lost_operator_mail_test.go:70: lost mails told = 0, want 1 (the one whose retries the restart ate; not the one that went out on a retry)
|
||||
|
||||
2 mutations, 0 not a red-proof
|
||||
== MUTATION: the give-up leaves only the log line
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestLostOperatorMail_GiveUpLeavesARow (0.04s)
|
||||
lost_operator_mail_test.go:106: after the retries gave up: event rows = [], want one that carries the lost mail's text
|
||||
== MUTATION: the four cases' events reach the household
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestR1383_SilentFailureEventsAreOperatorOnly (0.00s)
|
||||
lost_operator_mail_test.go:117: signed_job_refused must be operator-only
|
||||
|
||||
2 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,14 @@
|
||||
== MUTATION: a job that waits is never told
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s)
|
||||
signed_job_waits_test.go:35: events=0, want ONE (the old job, once): []
|
||||
== MUTATION: a fresh job is told at once
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s)
|
||||
signed_job_waits_test.go:35: events=2, want ONE (the old job, once): [cust-of-h1|signed_job_not_taken|warning|The signed job old for h1 is still in the queue after 3h0m0s: the box did not take it. A box asks for its jobs every 15
|
||||
== MUTATION: the job is told at every sweep
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s)
|
||||
signed_job_waits_test.go:35: events=2, want ONE (the old job, once): [cust-of-h1|signed_job_not_taken|warning|The signed job old for h1 is still in the queue after 3h0m0s: the box did not take it. A box asks for its jobs every 15
|
||||
|
||||
3 mutations, 0 not a red-proof
|
||||
@@ -332,7 +332,7 @@ stopping line that lies.
|
||||
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
|
||||
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
|
||||
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
|
||||
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
|
||||
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
|
||||
|
||||
|
||||
@@ -3,6 +3,18 @@
|
||||
- `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at
|
||||
every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change.
|
||||
- **Release 3 („the hub tells you”), built 2026-11-14 night, NOT deployed. Every new alarm is operator-only.**
|
||||
- **Four silent failures tell the operator (R-1383, D78, `09` §3 decision 299; `05` §16.6).** No new table.
|
||||
(1) `hub_daily_job_skipped`: `monitor.DailyLedger` keeps across a restart when `deadline-check` and
|
||||
`offsite-key-audit` last ran; a slot that passed while the hub was down is told once at the next start (the check
|
||||
is not run late). (2) `operator_mail_lost`: `Dispatcher.ReportLostOperatorMails` at a start sends ONE mail that
|
||||
carries every operator mail of the last 48 h that was never sent (its retry timers died with the restart, or all
|
||||
three retries failed); a give-up now also leaves an event row at once. (3) `signed_job_not_taken`: a signed job
|
||||
still in a box's queue after two hours, on the sweep. (4) from the box (agents of release 3): `signed_job_refused`
|
||||
— the agent's clearing `DELETE` says `outcome`, `reason` (closed list) and `op` in its query; and
|
||||
`guest_power_gave_up` / `guest_power_ok` from the host report's new `guest_power_gave_up` list. All in
|
||||
`notify.operatorOnlyEvents`. Tests `TestDailyLedger_*`, `TestPrevDailySlot_*`, `TestR1383_*`,
|
||||
`TestLostOperatorMail_*`, `TestSignedJobWait_*`, `TestSignedJobOutcome_*`, `TestGuestPowerGaveUp_*`; 31 mutations
|
||||
seen red (`documentation/audits/night-build-2026-11-14/B/redproof-hub-{daily-ledger,lost-mail,signed-job,box-told}.txt`).
|
||||
- **A box's clock against the hub's (R-1384, D79, `09` §3 decision 300).** The host report's new `sent_at` (the box's
|
||||
clock at the moment the agent sends; agents of release 3) is compared with the hub's clock at the moment the request
|
||||
arrives. More than one minute apart, in either direction → one operator-only `box_clock_wrong` (warning), told once;
|
||||
|
||||
+39
-1
@@ -362,6 +362,18 @@ func main() {
|
||||
)
|
||||
apiHandler.SetDispatcher(dispatcher)
|
||||
|
||||
// R-1383 (D78, decision 299): three of the four silent failures are the hub's own. hubEvent leaves a row AND
|
||||
// mails (the dispatcher keeps all of them operator-only). Pinned by r1383_wiring_test.go.
|
||||
hubEvent := func(cust, typ, sev, msg, details, src string) {
|
||||
if _, err := dataStore.SaveEvent(cust, typ, sev, msg, details, src); err != nil {
|
||||
logger.Printf("[WARN] %s event save FAILED: %v", typ, err)
|
||||
}
|
||||
dispatcher.ProcessEvent(cust, typ, sev, msg, details, src)
|
||||
}
|
||||
// (1) a daily check whose slot passed while the hub was down — told at this start.
|
||||
dailyLedger = monitor.NewDailyLedger(dataStore.OSAlarmRaised, dataStore.SetOSAlarmRaised, hubEvent, logger,
|
||||
"deadline-check", "offsite-key-audit")
|
||||
|
||||
// MAIL-HOLD (internal/mailhold): while <data_dir>/MAIL-HOLD exists the hub sends NO e-mail on any path —
|
||||
// dispatcher, /notify, app-mail relay. A hub started on a restored database mailed households within a minute
|
||||
// (2026-10-09 restore drill); the marker makes a restored copy start quiet. Lifted by the operator with
|
||||
@@ -915,6 +927,25 @@ func main() {
|
||||
// durable (events table), so a hub restart neither floods nor silently re-heals.
|
||||
// R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop.
|
||||
controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger)
|
||||
// R-1383 (2) an operator mail that was never sent — told at this start, in one mail (after the mail hold is wired).
|
||||
go dispatcher.ReportLostOperatorMails(time.Now())
|
||||
// R-1383 (3) a signed job a box never took — on the 60 s sweep.
|
||||
signedJobWaitChecker := monitor.NewSignedJobWaitChecker(
|
||||
func() ([]monitor.PendingSignedJob, error) {
|
||||
rows, err := dataStore.PendingSignedJobs()
|
||||
out := make([]monitor.PendingSignedJob, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
out = append(out, monitor.PendingSignedJob{HostID: r.HostID, JobID: r.JobID, CreatedAt: r.CreatedAt})
|
||||
}
|
||||
return out, err
|
||||
},
|
||||
func(hostID string) string {
|
||||
if h, err := dataStore.GetHost(hostID); err == nil && h != nil {
|
||||
return h.CustomerID
|
||||
}
|
||||
return ""
|
||||
},
|
||||
dataStore.OSAlarmRaised, dataStore.SetOSAlarmRaised, hubEvent, logger)
|
||||
// D53 (`08` §10): a held app reaches the OPERATOR — one reminder after the grace per reason, then weekly.
|
||||
var heldGrace time.Duration
|
||||
if v := cfg.Alerting.HeldAppGraceOverride; v != "" {
|
||||
@@ -957,7 +988,8 @@ func main() {
|
||||
restoreTestChecker.Check() // R-85: restore-test failure + per-tier staleness
|
||||
offsiteDeliveryChecker.Check()
|
||||
controllerSupervisorChecker.Check()
|
||||
heldAppsChecker.Check() // D53: throttled to one pass per 15 min
|
||||
signedJobWaitChecker.Check() // R-1383: a signed job a box did not take in two hours
|
||||
heldAppsChecker.Check() // D53: throttled to one pass per 15 min
|
||||
if goldenVouchChecker != nil {
|
||||
goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min
|
||||
}
|
||||
@@ -1079,6 +1111,10 @@ func loadConfig(path string, logger *log.Logger) *Config {
|
||||
return cfg
|
||||
}
|
||||
|
||||
// dailyLedger remembers across a restart when a watched daily check last ran (R-1383). nil until main wires it; a nil
|
||||
// ledger is safe to call.
|
||||
var dailyLedger *monitor.DailyLedger
|
||||
|
||||
// scheduleDaily runs fn once daily at the given "HH:MM" time in Europe/Budapest.
|
||||
// It blocks until ctx is cancelled.
|
||||
func scheduleDaily(ctx context.Context, name, timeStr string, fn func(), logger *log.Logger) {
|
||||
@@ -1088,6 +1124,7 @@ func scheduleDaily(ctx context.Context, name, timeStr string, fn func(), logger
|
||||
}
|
||||
|
||||
hour, min := parseHM(timeStr)
|
||||
dailyLedger.Started(name, time.Now(), hour, min, budapest) // R-1383: was the newest slot missed while the hub was down?
|
||||
|
||||
var last time.Time
|
||||
for {
|
||||
@@ -1101,6 +1138,7 @@ func scheduleDaily(ctx context.Context, name, timeStr string, fn func(), logger
|
||||
case <-time.After(delay):
|
||||
last = next
|
||||
fn()
|
||||
dailyLedger.Ran(name, next) // after fn: a hub that dies inside the check reads as "did not run"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1383 (D78): the three hub-side checks are components with seams; unit-green and never called is the inert-seam
|
||||
// defect (four instances in this project). The wiring is asserted from main.go's AST: the ledger is built and
|
||||
// assigned, the scheduler asks it at its start and tells it after each run, the lost-mail scan runs at a start, and
|
||||
// the signed-job check is on the sweep. COMPANION RED-PROOFS: audits/night-build-2026-11-14/B/redproof-hub-wiring.txt.
|
||||
func TestR1383_MainWiresTheSilentFailureChecks(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
seen := map[string]int{}
|
||||
inScheduler := map[string]int{}
|
||||
callName := func(c *ast.CallExpr) (recv, name string) {
|
||||
sel, ok := c.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return "", ""
|
||||
}
|
||||
if id, ok := sel.X.(*ast.Ident); ok {
|
||||
recv = id.Name
|
||||
}
|
||||
return recv, sel.Sel.Name
|
||||
}
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fn, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.AssignStmt:
|
||||
if len(node.Lhs) == 1 && len(node.Rhs) == 1 {
|
||||
if id, ok := node.Lhs[0].(*ast.Ident); ok && id.Name == "dailyLedger" {
|
||||
if c, ok := node.Rhs[0].(*ast.CallExpr); ok {
|
||||
if _, name := callName(c); name == "NewDailyLedger" {
|
||||
seen["ledger built"]++
|
||||
for _, a := range c.Args {
|
||||
if lit, ok := a.(*ast.BasicLit); ok && (lit.Value == `"deadline-check"` || lit.Value == `"offsite-key-audit"`) {
|
||||
seen["watched "+lit.Value]++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
recv, name := callName(node)
|
||||
switch {
|
||||
case recv == "dailyLedger" && fn.Name.Name == "scheduleDaily":
|
||||
inScheduler[name]++
|
||||
case recv == "dispatcher" && name == "ReportLostOperatorMails":
|
||||
seen["lost-mail scan"]++
|
||||
case recv == "signedJobWaitChecker" && name == "Check":
|
||||
seen["signed-job sweep"]++
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
for _, k := range []string{"ledger built", `watched "deadline-check"`, `watched "offsite-key-audit"`, "lost-mail scan", "signed-job sweep"} {
|
||||
if seen[k] != 1 {
|
||||
t.Errorf("main.go: %s — found %d time(s), want 1", k, seen[k])
|
||||
}
|
||||
}
|
||||
if inScheduler["Started"] != 1 || inScheduler["Ran"] != 1 {
|
||||
t.Errorf("scheduleDaily calls the ledger: Started %d, Ran %d — want 1 and 1", inScheduler["Started"], inScheduler["Ran"])
|
||||
}
|
||||
// the names the ledger watches are the names the scheduler is started with
|
||||
names := map[string]bool{}
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
if c, ok := n.(*ast.CallExpr); ok {
|
||||
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "scheduleDaily" && len(c.Args) > 1 {
|
||||
if lit, ok := c.Args[1].(*ast.BasicLit); ok {
|
||||
names[lit.Value] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
for _, n := range []string{`"deadline-check"`, `"offsite-key-audit"`} {
|
||||
if !names[n] {
|
||||
t.Errorf("no scheduleDaily job is named %s — the ledger watches a name nobody runs", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -778,6 +779,9 @@ type hostReportPayload struct {
|
||||
// ProxmoxTrust (R-1380, R-1396; `03` §12a): how the agent trusts Proxmox's certificate and, when something about it
|
||||
// is wrong, the CAUSE. nil = an agent without the stanza (older than release 2b).
|
||||
ProxmoxTrust *hostProxmoxTrust `json:"proxmox_trust"`
|
||||
// GuestPowerGaveUp (R-1383, D78): the guests the agent's guest-power watchdog stopped trying to start (three
|
||||
// failed starts). nil/empty = none, or an agent that does not report it.
|
||||
GuestPowerGaveUp []hostGuestPowerGaveUp `json:"guest_power_gave_up"`
|
||||
// SentAt (R-1384, D79): the box's clock at the moment the agent SENDS the report (stamped in the agent's
|
||||
// Client.Report, after the collect). "" = an agent that does not stamp it → the clock is not judged.
|
||||
// reported_at is NOT used for this: it is stamped when the collect starts, and a collect can take long.
|
||||
@@ -1018,6 +1022,13 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
|
||||
h.checkWGTunnel(hostID, custID, rep.Wireguard)
|
||||
h.checkProxmoxTrust(hostID, custID, rep.ProxmoxTrust)
|
||||
h.checkBoxClock(hostID, custID, rep.SentAt, received)
|
||||
runningGuests := map[int]bool{}
|
||||
for _, g := range rep.Guests {
|
||||
if g.Status == "running" {
|
||||
runningGuests[g.VMID] = true
|
||||
}
|
||||
}
|
||||
h.checkGuestPowerGaveUp(hostID, custID, rep.GuestPowerGaveUp, runningGuests)
|
||||
h.checkCrash(hostID, custID, body)
|
||||
h.checkDeliberateOff(hostID, custID, &rep)
|
||||
|
||||
@@ -2072,6 +2083,7 @@ func (h *Handler) handleDeleteJob(w http.ResponseWriter, r *http.Request, pathHo
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
h.noteSignedJobOutcome(pathHostID, jobID, r.URL.Query().Get("outcome"), r.URL.Query().Get("reason"), r.URL.Query().Get("op"))
|
||||
h.logger.Printf("[INFO] host %s cleared signed-op job %s (executed or rejected)", pathHostID, jobID)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
@@ -3246,6 +3258,120 @@ func (h *Handler) checkTunnel(hostID, custID, status, detail string) {
|
||||
}
|
||||
}
|
||||
|
||||
// The four silent failures, the two a BOX tells (R-1383, D78, `09` §3 decision 299; `08-alarm-ladder.md`).
|
||||
//
|
||||
// (a) signed_job_refused — the agent clears a signed job from its queue when it ran it AND when it refused it; until
|
||||
// now the hub could not tell the two apart („executed or rejected”), so an agent update that was refused left no
|
||||
// trace outside the box's journal (clock list CA09-4, CA09-7). An agent of release 3 says how the job ended in the
|
||||
// DELETE's query: outcome = done | rejected | failed | malformed, a reason from a CLOSED list, and the op's name.
|
||||
// Anything but `done` → one operator-only event. No outcome (an older agent) → nothing. The box's words never reach
|
||||
// a mail: an unknown reason is told as „other”, an op name is cut to [a-z0-9_].
|
||||
// (b) guest_power_gave_up — see checkGuestPowerGaveUp.
|
||||
// Pinned by TestSignedJobOutcome_* and TestGuestPowerGaveUp_*.
|
||||
const (
|
||||
eventSignedJobRefused = "signed_job_refused"
|
||||
eventGuestPowerGaveUp = "guest_power_gave_up"
|
||||
eventGuestPowerRunning = "guest_power_ok"
|
||||
)
|
||||
|
||||
// signedJobReasons is the closed list of the agent's reasons (signedjobs.reasonCode), each with the operator's
|
||||
// sentence. The two clock reasons name the clock, because a box with a wrong clock refuses EVERY job.
|
||||
var signedJobReasons = map[string]string{
|
||||
"expired": "its validity had run out when the box read it. If the job was fresh, the box's clock is AHEAD",
|
||||
"not_yet_valid": "it was not valid yet by the box's clock. The box's clock is BEHIND",
|
||||
"window_too_long": "its validity window is longer than the box allows",
|
||||
"replay": "the box saw this job's number before (a job runs once)",
|
||||
"bad_signature": "the signature did not verify",
|
||||
"unknown_signer": "the signing key is not one the box trusts",
|
||||
"target": "it names another box",
|
||||
"namespace": "it was signed for another use",
|
||||
"malformed": "the box could not read it",
|
||||
"role_denied": "the signing key may not sign this kind of job",
|
||||
"binding_mismatch": "it is signed for another action or device",
|
||||
"execution_failed": "it was accepted and its run FAILED on the box",
|
||||
}
|
||||
|
||||
var signedJobOpRe = regexp.MustCompile(`^[a-z0-9_]{1,40}$`)
|
||||
|
||||
func (h *Handler) noteSignedJobOutcome(hostID, jobID, outcome, reason, op string) {
|
||||
switch outcome {
|
||||
case "rejected", "failed", "malformed":
|
||||
default:
|
||||
return // "done", or an agent that does not say
|
||||
}
|
||||
key := "signed_job_refused:" + hostID + ":" + jobID
|
||||
if !h.store.OSAlarmRaised(key).IsZero() {
|
||||
return // the agent repeats a DELETE whose answer it did not get
|
||||
}
|
||||
_ = h.store.SetOSAlarmRaised(key, time.Now())
|
||||
text := signedJobReasons[reason]
|
||||
if text == "" {
|
||||
reason, text = "other", "the box gave a reason this hub does not know"
|
||||
}
|
||||
if !signedJobOpRe.MatchString(op) {
|
||||
op = "unknown"
|
||||
}
|
||||
custID := ""
|
||||
if host, err := h.store.GetHost(hostID); err == nil && host != nil {
|
||||
custID = host.CustomerID
|
||||
}
|
||||
what := "REFUSED"
|
||||
if outcome == "failed" {
|
||||
what = "ran and FAILED"
|
||||
}
|
||||
h.logger.Printf("[WARN] host %s signed job %s (%s): %s — %s", hostID, jobID, op, outcome, reason)
|
||||
msg := fmt.Sprintf("The signed job %s (%s) for %s was %s: %s. The job is gone from the queue and will not run again. Do: read the cause, then sign a new job.", jobID, op, hostID, what, text)
|
||||
details, _ := json.Marshal(map[string]any{"host_id": hostID, "job_id": jobID, "op": op, "outcome": outcome, "reason": reason})
|
||||
if _, err := h.store.SaveEvent(custID, eventSignedJobRefused, "error", msg, string(details), "hub"); err != nil {
|
||||
h.logger.Printf("[WARN] %s event save FAILED for %s: %v", eventSignedJobRefused, hostID, err)
|
||||
} else if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessEvent(custID, eventSignedJobRefused, "error", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
type hostGuestPowerGaveUp struct {
|
||||
VMID int `json:"vmid"`
|
||||
Since string `json:"since"`
|
||||
Attempts int `json:"attempts"`
|
||||
}
|
||||
|
||||
// checkGuestPowerGaveUp: the agent's guest-power watchdog starts a guest that must run and is stopped; after three
|
||||
// failed starts it stops trying, and until now it wrote one line in the host's journal and told nobody (clock list
|
||||
// CA07-7) — the household's apps are off and the only outside sign was „the box does not report”, which is not even
|
||||
// true here (the HOST reports). One operator-only `guest_power_gave_up` (error) per guest, told once for one down
|
||||
// period; one `guest_power_ok` when a later report shows that guest RUNNING. The agent's record is memory: after an
|
||||
// agent restart the entry is gone while the guest is still down — so „gone from the list” alone never clears.
|
||||
func (h *Handler) checkGuestPowerGaveUp(hostID, custID string, gave []hostGuestPowerGaveUp, running map[int]bool) {
|
||||
emit := func(typ, sev, msg string, details map[string]any) {
|
||||
dj, _ := json.Marshal(details)
|
||||
if _, err := h.store.SaveEvent(custID, typ, sev, msg, string(dj), "hub"); err != nil {
|
||||
h.logger.Printf("[WARN] %s event save FAILED for %s: %v", typ, hostID, err)
|
||||
} else if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(dj), "hub")
|
||||
}
|
||||
}
|
||||
key := func(vmid int) string { return fmt.Sprintf("guest_power:%s:%d", hostID, vmid) }
|
||||
for _, g := range gave {
|
||||
if g.VMID <= 0 || running[g.VMID] || !h.store.OSAlarmRaised(key(g.VMID)).IsZero() {
|
||||
continue
|
||||
}
|
||||
_ = h.store.SetOSAlarmRaised(key(g.VMID), time.Now())
|
||||
h.logger.Printf("[WARN] host %s: the guest-power watchdog gave up on guest %d (%d attempts)", hostID, g.VMID, g.Attempts)
|
||||
emit(eventGuestPowerGaveUp, "error",
|
||||
fmt.Sprintf("Guest %d on %s is STOPPED and the agent gave up starting it after %d failed attempts. The household's apps are off. The agent does not try again by itself. Do: read `journalctl -u felhom-agent` on the box for „guest-power”, start the guest, and read why it did not start.", g.VMID, hostID, g.Attempts),
|
||||
map[string]any{"host_id": hostID, "vmid": g.VMID, "attempts": g.Attempts, "since": g.Since})
|
||||
}
|
||||
for vmid := range running {
|
||||
if h.store.OSAlarmRaised(key(vmid)).IsZero() {
|
||||
continue
|
||||
}
|
||||
_ = h.store.SetOSAlarmRaised(key(vmid), time.Time{})
|
||||
h.logger.Printf("[INFO] host %s: guest %d runs again (the guest-power watchdog had given up)", hostID, vmid)
|
||||
emit(eventGuestPowerRunning, "info", fmt.Sprintf("Guest %d on %s runs again.", vmid, hostID),
|
||||
map[string]any{"host_id": hostID, "vmid": vmid})
|
||||
}
|
||||
}
|
||||
|
||||
// Box clock alarm (R-1384, D79, `09` §3 decision 300; `08-alarm-ladder.md`). The hub compares the box's clock at
|
||||
// the moment of SENDING (`sent_at`) with its own clock at the moment the request arrives. More than one minute apart,
|
||||
// in either direction → one operator-only `box_clock_wrong`; back within 30 seconds → one `box_clock_ok` (the gap
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
func r1383Events(t *testing.T, st *store.Store, typ string) []store.Event {
|
||||
t.Helper()
|
||||
evs, err := st.GetEventsByType("c1", typ, time.Now().Add(-time.Hour))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return evs
|
||||
}
|
||||
|
||||
// R-1383 (D78): an agent that clears a signed job says HOW it ended. A refused job (here: expired — the box's clock
|
||||
// is ahead, so an agent update never arrives) leaves one operator event that names the job, the op and the cause.
|
||||
// A job that ran, and an agent that says nothing (older), leave none. The stored effect is the event row; the queue
|
||||
// is emptied in every case. COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-box-told.txt).
|
||||
func TestSignedJobOutcome_RefusedIsToldDoneIsNot(t *testing.T) {
|
||||
h, st := trustSetup(t)
|
||||
del := func(job, query string) {
|
||||
t.Helper()
|
||||
if err := st.EnqueueSignedJob("h1", job, []byte("opaque")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rr := do(h, http.MethodDelete, "/hosts/h1/jobs/"+job+query, "HKEY", ""); rr.Code != 200 {
|
||||
t.Fatalf("DELETE %s: %d %s", job, rr.Code, rr.Body.String())
|
||||
}
|
||||
if n, _ := st.CountSignedJobs("h1"); n != 0 {
|
||||
t.Fatalf("job %s is still queued", job)
|
||||
}
|
||||
}
|
||||
del("ran", "?outcome=done&op=agent_update")
|
||||
del("old-agent", "")
|
||||
if evs := r1383Events(t, st, eventSignedJobRefused); len(evs) != 0 {
|
||||
t.Fatalf("a job that ran, or an agent that says nothing, raised %d event(s): %+v", len(evs), evs)
|
||||
}
|
||||
del("j-exp", "?outcome=rejected&reason=expired&op=agent_update")
|
||||
// the agent repeats a DELETE whose answer it lost: told once
|
||||
if rr := do(h, http.MethodDelete, "/hosts/h1/jobs/j-exp?outcome=rejected&reason=expired&op=agent_update", "HKEY", ""); rr.Code != 200 {
|
||||
t.Fatalf("repeat DELETE: %d", rr.Code)
|
||||
}
|
||||
evs := r1383Events(t, st, eventSignedJobRefused)
|
||||
if len(evs) != 1 {
|
||||
t.Fatalf("events=%d, want ONE for the refused job", len(evs))
|
||||
}
|
||||
for _, want := range []string{"j-exp", "agent_update", "h1", "REFUSED", "clock is AHEAD"} {
|
||||
if !strings.Contains(evs[0].Message, want) {
|
||||
t.Errorf("the event lacks %q: %s", want, evs[0].Message)
|
||||
}
|
||||
}
|
||||
if evs[0].Severity != "error" || !strings.Contains(evs[0].DetailsJSON, `"reason":"expired"`) {
|
||||
t.Errorf("event = %+v", evs[0])
|
||||
}
|
||||
// a failed run, and words the hub does not know: told, and the box's own text never reaches the mail
|
||||
del("j-fail", "?outcome=failed&reason=execution_failed&op=agent_config_update")
|
||||
del("j-odd", "?outcome=rejected&reason=%3Cscript%3E&op=%3Cb%3EX")
|
||||
evs = r1383Events(t, st, eventSignedJobRefused)
|
||||
if len(evs) != 3 {
|
||||
t.Fatalf("events=%d, want 3", len(evs))
|
||||
}
|
||||
all := ""
|
||||
for _, e := range evs {
|
||||
all += e.Message + e.DetailsJSON + "\n"
|
||||
}
|
||||
if !strings.Contains(all, "ran and FAILED") || !strings.Contains(all, `"reason":"other"`) || !strings.Contains(all, `"op":"unknown"`) {
|
||||
t.Errorf("the failed run or the unknown words are not told as designed:\n%s", all)
|
||||
}
|
||||
if strings.Contains(all, "script") || strings.Contains(all, "<b>") {
|
||||
t.Errorf("the box's own text reached an event:\n%s", all)
|
||||
}
|
||||
}
|
||||
|
||||
func gaveUpReport(guestStatus, stanza string) string {
|
||||
b := validReportBody("h1")
|
||||
b = strings.Replace(b, `{"vmid":100,"name":"acme","status":"running"`, `{"vmid":100,"name":"acme","status":"`+guestStatus+`"`, 1)
|
||||
if stanza == "" {
|
||||
return b
|
||||
}
|
||||
return b[:strings.LastIndex(b, "}")] + `,"guest_power_gave_up":` + stanza + "}"
|
||||
}
|
||||
|
||||
// R-1383 (D78): the guest-power watchdog gave up — the household's apps are off and the agent wrote one journal line.
|
||||
// One operator event per down period; the agent's list is memory, so the entry vanishing (an agent restart) while
|
||||
// the guest is still stopped must neither clear nor alarm a second time; the recovery is the guest RUNNING.
|
||||
func TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest(t *testing.T) {
|
||||
h, st := trustSetup(t)
|
||||
post := func(status, stanza string) {
|
||||
t.Helper()
|
||||
if rr := do(h, http.MethodPost, "/host-report", "HKEY", gaveUpReport(status, stanza)); rr.Code != 200 {
|
||||
t.Fatalf("report: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
counts := func() string {
|
||||
return fmt.Sprintf("gave_up=%d ok=%d", len(r1383Events(t, st, eventGuestPowerGaveUp)), len(r1383Events(t, st, eventGuestPowerRunning)))
|
||||
}
|
||||
post("running", "")
|
||||
post("stopped", "") // stopped, the watchdog still tries: nothing
|
||||
if c := counts(); c != "gave_up=0 ok=0" {
|
||||
t.Fatalf("before a give-up: %s", c)
|
||||
}
|
||||
stanza := `[{"vmid":100,"since":"2026-11-14T02:00:00Z","attempts":3}]`
|
||||
post("stopped", stanza)
|
||||
post("stopped", stanza)
|
||||
if c := counts(); c != "gave_up=1 ok=0" {
|
||||
t.Fatalf("two reports of the same give-up: %s, want ONE alarm", c)
|
||||
}
|
||||
e := r1383Events(t, st, eventGuestPowerGaveUp)[0]
|
||||
if e.Severity != "error" || !strings.Contains(e.Message, "Guest 100 on h1 is STOPPED") || !strings.Contains(e.Message, "3 failed attempts") {
|
||||
t.Errorf("alarm = %+v", e)
|
||||
}
|
||||
post("stopped", "") // the agent restarted: its list is empty, the guest is still down
|
||||
if c := counts(); c != "gave_up=1 ok=0" {
|
||||
t.Fatalf("the entry vanished while the guest is still stopped: %s — that is not a recovery", c)
|
||||
}
|
||||
post("stopped", `[{"vmid":100,"since":"2026-11-14T02:20:00Z","attempts":3}]`) // it gave up again: same down period
|
||||
if c := counts(); c != "gave_up=1 ok=0" {
|
||||
t.Fatalf("the second give-up of the same down period: %s", c)
|
||||
}
|
||||
post("running", "")
|
||||
post("running", "")
|
||||
if c := counts(); c != "gave_up=1 ok=1" {
|
||||
t.Fatalf("after the guest runs again: %s, want one recovery", c)
|
||||
}
|
||||
post("stopped", stanza) // a new down period is told again
|
||||
if c := counts(); c != "gave_up=2 ok=1" {
|
||||
t.Fatalf("a new down period: %s", c)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DailyLedger (R-1383, D78, `09` §3 decision 299; `05-hub-architecture.md` §16.5) remembers ACROSS A RESTART when a
|
||||
// hub daily check last ran, and tells the operator when a slot passed while the hub was down.
|
||||
//
|
||||
// Before it, „when did this job last run” lived in the scheduler's memory: a hub that was down at 05:00 skipped that
|
||||
// day's missed-backup check with no trace, and a missed backup was then seen one day late (clock list CH02-4, CH11-4,
|
||||
// CH16-4). The ledger does NOT run the missed check late — it tells. (The database snapshot and the prune make up for
|
||||
// a missed slot by themselves and are not watched.)
|
||||
//
|
||||
// The state is two times per job in the hub's settings (no new table): `daily_last:<name>` = the slot that last RAN
|
||||
// to its end, `daily_skip_told:<name>` = the newest missed slot that was told. Pinned by TestDailyLedger_*.
|
||||
type DailyLedger struct {
|
||||
get func(key string) time.Time
|
||||
set func(key string, t time.Time) error
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
watched map[string]bool
|
||||
}
|
||||
|
||||
// EventDailyJobSkipped is operator-only (notify.operatorOnlyEvents).
|
||||
const EventDailyJobSkipped = "hub_daily_job_skipped"
|
||||
|
||||
// NewDailyLedger: get/set are the hub's small time-by-key state (store.OSAlarmRaised / SetOSAlarmRaised); watched is
|
||||
// the list of job names the ledger speaks for.
|
||||
func NewDailyLedger(get func(string) time.Time, set func(string, time.Time) error, onEvent EventNotifyFunc, logger *log.Logger, watched ...string) *DailyLedger {
|
||||
w := map[string]bool{}
|
||||
for _, n := range watched {
|
||||
w[n] = true
|
||||
}
|
||||
return &DailyLedger{get: get, set: set, onEvent: onEvent, logger: logger, watched: w}
|
||||
}
|
||||
|
||||
// PrevDailySlot returns the newest HH:MM in loc that is not after now.
|
||||
func PrevDailySlot(now time.Time, hour, min int, loc *time.Location) time.Time {
|
||||
now = now.In(loc)
|
||||
for d := 0; ; d-- {
|
||||
p := time.Date(now.Year(), now.Month(), now.Day()+d, hour, min, 0, 0, loc)
|
||||
if !p.After(now) {
|
||||
return p
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Started is called once when the scheduler of `name` starts (a hub start). It tells the operator, once per missed
|
||||
// slot, when the newest slot before now did not run. A job the ledger has never seen is not judged: its watch begins
|
||||
// now.
|
||||
func (l *DailyLedger) Started(name string, now time.Time, hour, min int, loc *time.Location) {
|
||||
if l == nil || !l.watched[name] {
|
||||
return
|
||||
}
|
||||
last := l.get("daily_last:" + name)
|
||||
prev := PrevDailySlot(now, hour, min, loc)
|
||||
if last.IsZero() {
|
||||
// the first start with the ledger: nothing to compare with. The watch begins at the newest slot, so the next
|
||||
// missed one is seen.
|
||||
_ = l.set("daily_last:"+name, prev)
|
||||
l.logger.Printf("[INFO] daily ledger: %s is watched from now (no earlier record)", name)
|
||||
return
|
||||
}
|
||||
if !last.Before(prev) {
|
||||
return // the newest slot ran
|
||||
}
|
||||
if told := l.get("daily_skip_told:" + name); !told.Before(prev) {
|
||||
return // this missed slot was told at an earlier start
|
||||
}
|
||||
_ = l.set("daily_skip_told:"+name, prev)
|
||||
// how many slots passed: whole calendar days between the two slots
|
||||
missed := 0
|
||||
for s := prev; s.After(last) && missed < 400; s = time.Date(s.Year(), s.Month(), s.Day()-1, hour, min, 0, 0, loc) {
|
||||
missed++
|
||||
}
|
||||
l.logger.Printf("[WARN] daily ledger: %s did not run at %s — the hub was not running then (last run %s; %d slot(s) missed)",
|
||||
name, prev.Format("2006-01-02 15:04 MST"), last.In(loc).Format("2006-01-02 15:04 MST"), missed)
|
||||
msg := fmt.Sprintf("The hub's daily check „%s” did NOT run at %s: the hub was not running at that minute. It last ran at %s (%d slot(s) missed). The hub does not run it late. %s",
|
||||
name, prev.Format("2006-01-02 15:04 MST"), last.In(loc).Format("2006-01-02 15:04 MST"), missed, dailySkipAdvice(name))
|
||||
details, _ := json.Marshal(map[string]any{"job": name, "missed_slot": prev.UTC().Format(time.RFC3339),
|
||||
"last_run": last.UTC().Format(time.RFC3339), "missed": missed})
|
||||
if l.onEvent != nil {
|
||||
l.onEvent("", EventDailyJobSkipped, "warning", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// Ran records that the slot ran to its end.
|
||||
func (l *DailyLedger) Ran(name string, slot time.Time) {
|
||||
if l == nil || !l.watched[name] {
|
||||
return
|
||||
}
|
||||
if err := l.set("daily_last:"+name, slot); err != nil {
|
||||
l.logger.Printf("[WARN] daily ledger: could not record the run of %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
func dailySkipAdvice(name string) string {
|
||||
switch name {
|
||||
case "deadline-check":
|
||||
return "What it means: a backup that was missed last night is told one day late. Do: read the Backup card of each box today."
|
||||
case "offsite-key-audit":
|
||||
return "What it means: the off-site keys were not audited that day. The next slot audits them; no hand step."
|
||||
}
|
||||
return "The next slot runs it."
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type ledgerFix struct {
|
||||
state map[string]time.Time
|
||||
events []string
|
||||
l *DailyLedger
|
||||
}
|
||||
|
||||
func newLedgerFix(watched ...string) *ledgerFix {
|
||||
f := &ledgerFix{state: map[string]time.Time{}}
|
||||
f.l = NewDailyLedger(
|
||||
func(k string) time.Time { return f.state[k] },
|
||||
func(k string, t time.Time) error { f.state[k] = t; return nil },
|
||||
func(cust, typ, sev, msg, details, src string) {
|
||||
f.events = append(f.events, cust+"|"+typ+"|"+sev+"|"+src+"|"+msg+"|"+details)
|
||||
},
|
||||
log.New(io.Discard, "", 0), watched...)
|
||||
return f
|
||||
}
|
||||
|
||||
func bud(t *testing.T) *time.Location {
|
||||
t.Helper()
|
||||
loc, err := time.LoadLocation("Europe/Budapest")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return loc
|
||||
}
|
||||
|
||||
// R-1383 (D78), the case the ruling names: the hub was DOWN at 05:00. Before the ledger the scheduler's „last run”
|
||||
// lived in memory, so the next start knew nothing. The stored effect asserted here: ONE operator event that names the
|
||||
// job and the missed slot — and none for a slot that ran, none twice for the same slot, none for a job nobody watches.
|
||||
// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-daily-ledger.txt).
|
||||
func TestDailyLedger_HubDownAtTheSlotIsTold(t *testing.T) {
|
||||
loc := bud(t)
|
||||
at := func(d, h, m int) time.Time { return time.Date(2026, 11, d, h, m, 0, 0, loc) }
|
||||
f := newLedgerFix("deadline-check")
|
||||
|
||||
// day 10: the first start ever at 04:00 — nothing to compare with, no word
|
||||
f.l.Started("deadline-check", at(10, 4, 0), 5, 0, loc)
|
||||
// 05:00 runs
|
||||
f.l.Ran("deadline-check", at(10, 5, 0))
|
||||
// a restart the same day at 09:00: the 05:00 slot ran → no word
|
||||
f.l.Started("deadline-check", at(10, 9, 0), 5, 0, loc)
|
||||
if len(f.events) != 0 {
|
||||
t.Fatalf("a slot that ran was told as skipped: %v", f.events)
|
||||
}
|
||||
// the hub goes down on day 10 at 23:00 and starts again on day 11 at 05:00:30 — the 05:00 slot passed
|
||||
f.l.Started("deadline-check", at(11, 5, 0).Add(30*time.Second), 5, 0, loc)
|
||||
if len(f.events) != 1 {
|
||||
t.Fatalf("events=%d, want ONE for the slot the hub was down at: %v", len(f.events), f.events)
|
||||
}
|
||||
e := f.events[0]
|
||||
for _, want := range []string{"|hub_daily_job_skipped|warning|hub|", "deadline-check", "2026-11-11 05:00", "2026-11-10 05:00", "1 slot(s) missed", `"missed_slot":"2026-11-11T04:00:00Z"`} {
|
||||
if !strings.Contains(e, want) {
|
||||
t.Errorf("the event lacks %q:\n%s", want, e)
|
||||
}
|
||||
}
|
||||
if !strings.HasPrefix(e, "|") {
|
||||
t.Errorf("a hub-wide event must carry no customer: %s", e)
|
||||
}
|
||||
// a second restart the same morning: told once
|
||||
f.l.Started("deadline-check", at(11, 6, 0), 5, 0, loc)
|
||||
if len(f.events) != 1 {
|
||||
t.Fatalf("the same missed slot was told %d times", len(f.events))
|
||||
}
|
||||
// day 12 runs; a restart after it is quiet; then two days down → told again, and it counts 2
|
||||
f.l.Ran("deadline-check", at(12, 5, 0))
|
||||
f.l.Started("deadline-check", at(12, 12, 0), 5, 0, loc)
|
||||
if len(f.events) != 1 {
|
||||
t.Fatalf("a restart after a slot that ran was told: %v", f.events[1:])
|
||||
}
|
||||
f.l.Started("deadline-check", at(14, 8, 0), 5, 0, loc)
|
||||
if len(f.events) != 2 || !strings.Contains(f.events[1], "2 slot(s) missed") || !strings.Contains(f.events[1], "2026-11-14 05:00") {
|
||||
t.Fatalf("two days down: %v", f.events[1:])
|
||||
}
|
||||
}
|
||||
|
||||
// A job that is not on the watch list (the snapshot and the prune make up for a missed slot by themselves) is never
|
||||
// told and leaves no state. And a nil ledger is safe: the scheduler calls it without a check.
|
||||
func TestDailyLedger_UnwatchedJobAndNilLedger(t *testing.T) {
|
||||
loc := bud(t)
|
||||
f := newLedgerFix("deadline-check")
|
||||
f.l.Ran("prune", time.Date(2026, 11, 1, 3, 0, 0, 0, loc))
|
||||
f.l.Started("prune", time.Date(2026, 11, 9, 9, 0, 0, 0, loc), 3, 0, loc)
|
||||
if len(f.events) != 0 || len(f.state) != 0 {
|
||||
t.Fatalf("an unwatched job: events=%v state=%v", f.events, f.state)
|
||||
}
|
||||
var nl *DailyLedger
|
||||
nl.Started("deadline-check", time.Now(), 5, 0, loc)
|
||||
nl.Ran("deadline-check", time.Now())
|
||||
}
|
||||
|
||||
// The slot is a wall-clock time in Budapest: on the two days the clocks change it is still „the newest 05:00”.
|
||||
func TestPrevDailySlot_HourChangeDays(t *testing.T) {
|
||||
loc := bud(t)
|
||||
for _, c := range []struct{ now, want string }{
|
||||
{"2026-10-25T04:30:00+01:00", "2026-10-24T05:00:00+02:00"}, // 25-hour day, before the slot
|
||||
{"2026-10-25T05:00:00+01:00", "2026-10-25T05:00:00+01:00"}, // at the slot: it is the slot
|
||||
{"2027-03-28T12:00:00+02:00", "2027-03-28T05:00:00+02:00"}, // 23-hour day
|
||||
} {
|
||||
now, _ := time.Parse(time.RFC3339, c.now)
|
||||
want, _ := time.Parse(time.RFC3339, c.want)
|
||||
if got := PrevDailySlot(now, 5, 0, loc); !got.Equal(want) {
|
||||
t.Errorf("PrevDailySlot(%s) = %s, want %s", c.now, got.Format(time.RFC3339), c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
)
|
||||
|
||||
// SignedJobWaitChecker (R-1383, D78, `09` §3 decision 299) tells the operator when a signed job is still in a box's
|
||||
// queue long after it was put there — the box did not take it, so (for one) an agent update never arrives. Before it,
|
||||
// a job that ran out left no trace anywhere (clock list CA09-4, CA09-7).
|
||||
//
|
||||
// The hub does not open a signed job (it stores and serves bytes), so it cannot read the job's own end time. It
|
||||
// judges by AGE in the queue: a box asks every 15 minutes at the latest, so a job that waits two hours was not taken.
|
||||
// One event per job, told once (the hub's time-by-key state); nothing is removed from the queue. The other half — a
|
||||
// job the box took and REFUSED — is the agent's word at the moment it clears the job (api.handleDeleteJob).
|
||||
// Pinned by TestSignedJobWait_*.
|
||||
type SignedJobWaitChecker struct {
|
||||
pending func() ([]PendingSignedJob, error)
|
||||
customer func(hostID string) string
|
||||
told func(key string) time.Time
|
||||
setTold func(key string, t time.Time) error
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
now func() time.Time
|
||||
after time.Duration
|
||||
}
|
||||
|
||||
// PendingSignedJob is one job in a box's queue (store.PendingSignedJobs).
|
||||
type PendingSignedJob struct {
|
||||
HostID string
|
||||
JobID string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
const (
|
||||
// EventSignedJobNotTaken is operator-only (notify.operatorOnlyEvents).
|
||||
EventSignedJobNotTaken = "signed_job_not_taken"
|
||||
signedJobWaitsTooLong = 2 * time.Hour
|
||||
)
|
||||
|
||||
func NewSignedJobWaitChecker(pending func() ([]PendingSignedJob, error), customer func(string) string,
|
||||
told func(string) time.Time, setTold func(string, time.Time) error, onEvent EventNotifyFunc, logger *log.Logger) *SignedJobWaitChecker {
|
||||
return &SignedJobWaitChecker{pending: pending, customer: customer, told: told, setTold: setTold, onEvent: onEvent,
|
||||
logger: logger, now: time.Now, after: signedJobWaitsTooLong}
|
||||
}
|
||||
|
||||
// Check runs on the hub's 60 s sweep.
|
||||
func (c *SignedJobWaitChecker) Check() {
|
||||
jobs, err := c.pending()
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] signed-job wait check: cannot read the queue: %v", err)
|
||||
return
|
||||
}
|
||||
now := c.now()
|
||||
for _, j := range jobs {
|
||||
if j.CreatedAt.IsZero() || now.Sub(j.CreatedAt) < c.after {
|
||||
continue
|
||||
}
|
||||
key := "signed_job_wait:" + j.HostID + ":" + j.JobID
|
||||
if !c.told(key).IsZero() {
|
||||
continue
|
||||
}
|
||||
_ = c.setTold(key, now)
|
||||
waited := now.Sub(j.CreatedAt).Round(time.Minute)
|
||||
c.logger.Printf("[WARN] signed job %s for host %s was not taken (in the queue for %s)", j.JobID, j.HostID, waited)
|
||||
msg := fmt.Sprintf("The signed job %s for %s is still in the queue after %s: the box did not take it. A box asks for its jobs every 15 minutes, so the box is off, cannot reach the hub, or its agent does not run. The job's validity has probably run out. Do: read why the box does not report, then sign a new job.",
|
||||
j.JobID, j.HostID, waited)
|
||||
details, _ := json.Marshal(map[string]any{"host_id": j.HostID, "job_id": j.JobID,
|
||||
"queued_at": j.CreatedAt.UTC().Format(time.RFC3339), "waited_minutes": int(waited / time.Minute)})
|
||||
if c.onEvent != nil {
|
||||
c.onEvent(c.customer(j.HostID), EventSignedJobNotTaken, "warning", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1383 (D78): a signed job the box never took is told ONCE, after two hours in the queue, to the box's customer
|
||||
// record (the dispatcher keeps it operator-only). A fresh job, and a job that left the queue, raise nothing.
|
||||
// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-signed-job.txt).
|
||||
func TestSignedJobWait_OldJobToldOnceFreshJobNot(t *testing.T) {
|
||||
now := time.Date(2026, 11, 14, 12, 0, 0, 0, time.UTC)
|
||||
queue := []PendingSignedJob{
|
||||
{HostID: "h1", JobID: "old", CreatedAt: now.Add(-3 * time.Hour)},
|
||||
{HostID: "h1", JobID: "fresh", CreatedAt: now.Add(-119 * time.Minute)},
|
||||
{HostID: "h2", JobID: "nodate"},
|
||||
}
|
||||
state := map[string]time.Time{}
|
||||
var events []string
|
||||
c := NewSignedJobWaitChecker(
|
||||
func() ([]PendingSignedJob, error) { return queue, nil },
|
||||
func(h string) string { return "cust-of-" + h },
|
||||
func(k string) time.Time { return state[k] },
|
||||
func(k string, t time.Time) error { state[k] = t; return nil },
|
||||
func(cust, typ, sev, msg, details, src string) {
|
||||
events = append(events, cust+"|"+typ+"|"+sev+"|"+msg+"|"+details)
|
||||
}, log.New(io.Discard, "", 0))
|
||||
c.now = func() time.Time { return now }
|
||||
c.Check()
|
||||
c.Check()
|
||||
if len(events) != 1 {
|
||||
t.Fatalf("events=%d, want ONE (the old job, once): %v", len(events), events)
|
||||
}
|
||||
for _, want := range []string{"cust-of-h1|signed_job_not_taken|warning|", "job old ", "h1", "3h0m0s", `"job_id":"old"`, `"waited_minutes":180`} {
|
||||
if !strings.Contains(events[0], want) {
|
||||
t.Errorf("the event lacks %q: %s", want, events[0])
|
||||
}
|
||||
}
|
||||
// two minutes later the second job crosses the line: it is told, the first is not told again
|
||||
now = now.Add(2 * time.Minute)
|
||||
c.Check()
|
||||
if len(events) != 2 || !strings.Contains(events[1], `"job_id":"fresh"`) {
|
||||
t.Fatalf("after the second job crossed two hours: %v", events)
|
||||
}
|
||||
// the box takes both: nothing more, ever
|
||||
queue = nil
|
||||
now = now.Add(24 * time.Hour)
|
||||
c.Check()
|
||||
if len(events) != 2 {
|
||||
t.Fatalf("an empty queue raised an event: %v", events[2:])
|
||||
}
|
||||
}
|
||||
@@ -99,6 +99,15 @@ func NewDispatcher(s *store.Store, resendAPIKey, fromEmail, operatorEmail string
|
||||
func (d *Dispatcher) retryOperatorEmail(customerID, eventType, severity, message, subject, body string, headers map[string]string, attempt int) {
|
||||
if attempt >= len(operatorRetryDelays) {
|
||||
d.logger.Printf("[ERROR] Operator email for %s/%s GAVE UP after %d retries — it was never sent", customerID, eventType, attempt)
|
||||
// R-1383: the give-up leaves a ROW, not only a log line (no mail — the mail service is what failed). The next
|
||||
// hub start tells it by mail (ReportLostOperatorMails).
|
||||
if eventType != EventOperatorMailLost {
|
||||
dj, _ := json.Marshal(map[string]any{"customer_id": customerID, "event_type": eventType, "retries": attempt})
|
||||
if _, err := d.store.SaveEvent("", EventOperatorMailLost, "error",
|
||||
fmt.Sprintf("The operator mail for %s/%s was never sent: the first send and %d retries failed. Its text: %s", customerID, eventType, attempt, message), string(dj), "hub"); err != nil {
|
||||
d.logger.Printf("[WARN] %s event save FAILED: %v", EventOperatorMailLost, err)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
d.afterFn(operatorRetryDelays[attempt], func() {
|
||||
@@ -844,6 +853,13 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// R-1396 / R-1380 (hub v0.182.0): why a box's agent does not trust Proxmox's certificate, and its recovery — a box
|
||||
// fact only the operator can act on. Hub-origin only (minted from the host report's `proxmox_trust` stanza).
|
||||
"proxmox_trust_problem": true,
|
||||
// R-1383 (D78): the four silent failures — hub facts and box facts only the operator can act on.
|
||||
"hub_daily_job_skipped": true,
|
||||
"signed_job_not_taken": true,
|
||||
"signed_job_refused": true,
|
||||
"operator_mail_lost": true,
|
||||
"guest_power_gave_up": true,
|
||||
"guest_power_ok": true,
|
||||
"box_clock_wrong": true, // R-1384: a box's clock against the hub's — the operator's to read
|
||||
"box_clock_ok": true,
|
||||
"proxmox_trust_ok": true,
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// EventOperatorMailLost (R-1383, D78, `09` §3 decision 299) is operator-only.
|
||||
const EventOperatorMailLost = "operator_mail_lost"
|
||||
|
||||
// lostMailLookBack bounds the start-up scan: an alarm older than two days is no longer news.
|
||||
const lostMailLookBack = 48 * time.Hour
|
||||
|
||||
// ReportLostOperatorMails runs ONCE at a hub start (cmd/hub/main.go). The retries of a failed operator mail are
|
||||
// timers in memory (operatorRetryDelays, 21 minutes in all): a hub restart inside them lost the mail, and after the
|
||||
// last failed retry nothing sent it either (clock list CH04-4, CH04-7). At a start no timer is alive, so every
|
||||
// operator mail whose newest delivery record is `failed` was never sent. They are told in ONE operator mail that
|
||||
// carries each lost mail's own sentence, and in one event row. The scan's own time is kept (hub settings), so a second
|
||||
// restart does not tell the same mails again.
|
||||
//
|
||||
// The limit, said plainly: a mail that gave up while the hub KEEPS running is told only at the next hub start (it
|
||||
// leaves an event row at once — see retryOperatorEmail). And this summary is an operator mail like any other: when the
|
||||
// mail service is still down it is retried three times and can be lost too — the next start then lists it.
|
||||
// Returns the number of lost mails told. Pinned by TestLostOperatorMail_*.
|
||||
func (d *Dispatcher) ReportLostOperatorMails(now time.Time) int {
|
||||
since := now.Add(-lostMailLookBack)
|
||||
if mark := d.store.OSAlarmRaised("operator_mail_lost_scan"); mark.After(since) {
|
||||
since = mark
|
||||
}
|
||||
lost, err := d.store.LostOperatorMails(since)
|
||||
if err != nil {
|
||||
d.logger.Printf("[WARN] lost operator mail scan failed: %v", err)
|
||||
return 0
|
||||
}
|
||||
if err := d.store.SetOSAlarmRaised("operator_mail_lost_scan", now); err != nil {
|
||||
d.logger.Printf("[WARN] lost operator mail scan: could not record the scan time: %v", err)
|
||||
}
|
||||
if len(lost) == 0 {
|
||||
d.logger.Printf("[INFO] lost operator mail scan: none since %s", since.UTC().Format(time.RFC3339))
|
||||
return 0
|
||||
}
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "%d operator mail(s) were NEVER sent: the send failed and the hub restarted or gave up before a retry worked. Each one, with its own text:", len(lost))
|
||||
type item struct {
|
||||
At, Customer, Event, Severity string
|
||||
}
|
||||
var items []item
|
||||
for i, m := range lost {
|
||||
if i == 20 {
|
||||
fmt.Fprintf(&b, "\n… and %d more (the hub's notification log holds them, status „failed”).", len(lost)-20)
|
||||
break
|
||||
}
|
||||
msg := m.Message
|
||||
if len(msg) > 400 {
|
||||
msg = msg[:400] + "…"
|
||||
}
|
||||
cust := m.CustomerID
|
||||
if cust == "" {
|
||||
cust = "(hub)"
|
||||
}
|
||||
fmt.Fprintf(&b, "\n- %s · %s · %s (%s): %s", m.CreatedAt.UTC().Format("2006-01-02 15:04Z"), cust, m.EventType, m.Severity, msg)
|
||||
items = append(items, item{m.CreatedAt.UTC().Format(time.RFC3339), m.CustomerID, m.EventType, m.Severity})
|
||||
}
|
||||
d.logger.Printf("[ERROR] lost operator mail scan: %d operator mail(s) were never sent — telling the operator in one mail", len(lost))
|
||||
details, _ := json.Marshal(map[string]any{"count": len(lost), "since": since.UTC().Format(time.RFC3339), "lost": items})
|
||||
if _, err := d.store.SaveEvent("", EventOperatorMailLost, "error", b.String(), string(details), "hub"); err != nil {
|
||||
d.logger.Printf("[WARN] %s event save FAILED: %v", EventOperatorMailLost, err)
|
||||
}
|
||||
d.ProcessEvent("", EventOperatorMailLost, "error", b.String(), string(details), "hub")
|
||||
return len(lost)
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type sentMail struct{ to, subject, body string }
|
||||
|
||||
func lostMailDispatcher(t *testing.T) (*Dispatcher, *[]sentMail, *bool) {
|
||||
t.Helper()
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
var mu sync.Mutex
|
||||
sent, down := &[]sentMail{}, new(bool)
|
||||
d.sendEmailFn = func(to, subject, body string, _ map[string]string) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if *down {
|
||||
return errors.New("context deadline exceeded")
|
||||
}
|
||||
*sent = append(*sent, sentMail{to, subject, body})
|
||||
return nil
|
||||
}
|
||||
return d, sent, down
|
||||
}
|
||||
|
||||
// R-1383 (D78), the case the ruling names: an operator mail failed, and the hub RESTARTED before a retry worked — the
|
||||
// retry timers are memory, so nothing sent it. The stored effect asserted: at the next start ONE operator mail goes
|
||||
// out that carries the lost mail's own sentence, one event row is left, and a second start tells nothing again.
|
||||
// A mail that failed and then went out on a retry is NOT lost. No real mail service: sendEmailFn is the repo's fake.
|
||||
// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-lost-mail.txt).
|
||||
func TestLostOperatorMail_ToldAtTheNextStart(t *testing.T) {
|
||||
d, sent, down := lostMailDispatcher(t)
|
||||
// 1. a mail that fails, and the "restart" eats its retry timers
|
||||
*down = true
|
||||
d.afterFn = func(time.Duration, func()) {} // the timer never fires: the hub restarted
|
||||
d.processOperator("c1", "whole_guest_backup_failed", "error", "Whole-guest backup FAILED on demo-hp", "", "box")
|
||||
// 2. a second alarm of the same kind inside the cooldown is suppressed — that row must not hide the failed one
|
||||
d.processOperator("c1", "whole_guest_backup_failed", "error", "Whole-guest backup FAILED on demo-hp", "", "box")
|
||||
// 3. a mail that fails once and goes out on its first retry — NOT lost
|
||||
tries := 0
|
||||
d.afterFn = func(_ time.Duration, f func()) { f() }
|
||||
d.sendEmailFn = func(to, subject, body string, _ map[string]string) error {
|
||||
tries++
|
||||
if tries == 1 {
|
||||
return errors.New("timeout")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
d.processOperator("c1", "tunnel_down", "error", "The tunnel of demo-hp is NOT running", "", "hub")
|
||||
if tries != 2 {
|
||||
t.Fatalf("setup: the retried mail made %d tries, want 2", tries)
|
||||
}
|
||||
|
||||
// the hub starts again: a new dispatcher on the same database, the mail service works
|
||||
d2 := NewDispatcher(d.store, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
var got []sentMail
|
||||
d2.sendEmailFn = func(to, subject, body string, _ map[string]string) error {
|
||||
got = append(got, sentMail{to, subject, body})
|
||||
return nil
|
||||
}
|
||||
now := time.Now().Add(time.Minute)
|
||||
if n := d2.ReportLostOperatorMails(now); n != 1 {
|
||||
t.Fatalf("lost mails told = %d, want 1 (the one whose retries the restart ate; not the one that went out on a retry)", n)
|
||||
}
|
||||
if len(got) != 1 || got[0].to != "op@felhom.eu" {
|
||||
t.Fatalf("mails sent at the start = %+v, want ONE to the operator", got)
|
||||
}
|
||||
for _, want := range []string{"Whole-guest backup FAILED on demo-hp", "whole_guest_backup_failed", "c1"} {
|
||||
if !strings.Contains(got[0].body, want) {
|
||||
t.Errorf("the summary mail lacks %q:\n%s", want, got[0].body)
|
||||
}
|
||||
}
|
||||
if strings.Contains(got[0].body, "The tunnel of demo-hp") {
|
||||
t.Errorf("a mail that WAS sent on a retry is listed as lost:\n%s", got[0].body)
|
||||
}
|
||||
evs, _ := d.store.GetEventsByType("", EventOperatorMailLost, time.Now().Add(-time.Hour))
|
||||
if len(evs) != 1 || evs[0].Severity != "error" || !strings.Contains(evs[0].DetailsJSON, `"count":1`) {
|
||||
t.Fatalf("event rows = %+v, want one operator_mail_lost row", evs)
|
||||
}
|
||||
// a second restart: the same mail is not told again
|
||||
if n := d2.ReportLostOperatorMails(now.Add(time.Minute)); n != 0 || len(got) != 1 {
|
||||
t.Fatalf("a second start told %d again (mails now %d)", n, len(got))
|
||||
}
|
||||
_ = sent
|
||||
if !operatorOnlyEvents[EventOperatorMailLost] {
|
||||
t.Errorf("%s is not operator-only", EventOperatorMailLost)
|
||||
}
|
||||
}
|
||||
|
||||
// The other half (clock list CH04-7): the hub keeps running and all three retries fail. A log line was the only
|
||||
// trace; now an event row says which mail was never sent. (No mail: the mail service is what failed.)
|
||||
func TestLostOperatorMail_GiveUpLeavesARow(t *testing.T) {
|
||||
d, sent, down := lostMailDispatcher(t)
|
||||
*down = true
|
||||
d.afterFn = func(_ time.Duration, f func()) { f() }
|
||||
d.processOperator("c1", "whole_guest_backup_failed", "error", "Whole-guest backup FAILED on demo-hp", "", "box")
|
||||
evs, _ := d.store.GetEventsByType("", EventOperatorMailLost, time.Now().Add(-time.Hour))
|
||||
if len(evs) != 1 || !strings.Contains(evs[0].Message, "Whole-guest backup FAILED on demo-hp") || !strings.Contains(evs[0].Message, "3 retries") {
|
||||
t.Fatalf("after the retries gave up: event rows = %+v, want one that carries the lost mail's text", evs)
|
||||
}
|
||||
if len(*sent) != 0 {
|
||||
t.Fatalf("a mail went out while the mail service was down: %+v", *sent)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383: the other three events of the four cases are operator-only too.
|
||||
func TestR1383_SilentFailureEventsAreOperatorOnly(t *testing.T) {
|
||||
for _, e := range []string{"hub_daily_job_skipped", "signed_job_not_taken", "signed_job_refused", "operator_mail_lost", "guest_power_gave_up"} {
|
||||
if !operatorOnlyEvents[e] {
|
||||
t.Errorf("%s must be operator-only", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package store
|
||||
|
||||
import "time"
|
||||
|
||||
// PendingSignedJobRow is one job in a box's signed-job queue, with the time it was put there (R-1383).
|
||||
type PendingSignedJobRow struct {
|
||||
HostID string
|
||||
JobID string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
// PendingSignedJobs lists EVERY box's waiting signed jobs, oldest first (the hub's „not taken” check, R-1383). The
|
||||
// blob is not read: the hub does not open a signed job.
|
||||
func (s *Store) PendingSignedJobs() ([]PendingSignedJobRow, error) {
|
||||
rows, err := s.db.Query(`SELECT host_id, job_id, created_at FROM signed_jobs ORDER BY id ASC`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []PendingSignedJobRow
|
||||
for rows.Next() {
|
||||
var r PendingSignedJobRow
|
||||
var at string
|
||||
if err := rows.Scan(&r.HostID, &r.JobID, &at); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.CreatedAt = parseSQLiteTime(at)
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// LostOperatorMail is an operator mail whose LAST delivery record is `failed` (R-1383).
|
||||
type LostOperatorMail struct {
|
||||
CustomerID string
|
||||
EventType string
|
||||
Severity string
|
||||
Message string
|
||||
Error string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
// LostOperatorMails returns the operator mails recorded after `since` whose newest delivery record (sent, failed or
|
||||
// held — a cooldown's `suppressed` row is not a delivery) is `failed`: the mail was never sent. Read at a hub START,
|
||||
// when no retry timer can be alive, so every such row is a mail nothing will send (R-1383; clock list CH04-4, CH04-7).
|
||||
func (s *Store) LostOperatorMails(since time.Time) ([]LostOperatorMail, error) {
|
||||
rows, err := s.db.Query(`
|
||||
SELECT n.customer_id, n.event_type, n.severity, n.message, COALESCE(n.error_message, ''), n.created_at
|
||||
FROM notification_log n
|
||||
WHERE n.channel = 'operator' AND n.status = 'failed' AND n.created_at > ?
|
||||
AND n.id = (SELECT MAX(m.id) FROM notification_log m
|
||||
WHERE m.channel = 'operator' AND m.customer_id = n.customer_id AND m.event_type = n.event_type
|
||||
AND m.message = n.message AND m.status IN ('sent', 'failed', 'held'))
|
||||
ORDER BY n.id ASC`, since.UTC().Format("2006-01-02 15:04:05"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []LostOperatorMail
|
||||
for rows.Next() {
|
||||
var m LostOperatorMail
|
||||
var at string
|
||||
if err := rows.Scan(&m.CustomerID, &m.EventType, &m.Severity, &m.Message, &m.Error, &at); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m.CreatedAt = parseSQLiteTime(at)
|
||||
out = append(out, m)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
Reference in new issue
Block a user