diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index df234b36..7a6f549f 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -645,6 +645,24 @@ never „now + 24 h", and the slot that ran last never runs again. On a 25-hour arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for 2026-10-25 and 2027-03-28. +### 16.6 A job that does not run, gives up, or loses its state at a restart tells the operator [FACT — built, not released: hub `main` after v0.181.0 and agent `main` after 0.174.0, 2026-11-14; D78, `09` §3 decision 299, R-1383] + +Four cases, the four the ruling names. Each leaves an event row and mails the operator only +(`notify.operatorOnlyEvents`). None needs a new table: the state is the hub's time-by-key settings +(`OSAlarmRaised`), the notification log and the signed-job queue. + +| Case | Who sees it | How | Event | What it does NOT do | +|---|---|---|---|---| +| A hub daily check's slot passed while the hub was down | the hub, at its next start | `monitor.DailyLedger` keeps „the slot that last ran to its end” per watched job (`deadline-check`, `offsite-key-audit`) across a restart; `scheduleDaily` asks it at its start and tells it after each run | `hub_daily_job_skipped` (warning), once per missed slot | it does not run the missed check late. The snapshot and the prune are not watched (they make up for a missed slot by themselves). A job it has never seen is watched from the first start, not judged | +| An operator mail was never sent | the hub, at its next start | `Dispatcher.ReportLostOperatorMails`: at a start no retry timer is alive, so every operator mail of the last 48 hours whose newest delivery record is `failed` was never sent. ONE mail carries each lost mail's own sentence; the scan's time is kept, so a second restart does not repeat it | `operator_mail_lost` (error) | a mail that gave up while the hub keeps running leaves an event row at once and is MAILED only at the next start. The summary is a mail like any other and can be lost too; the next start lists it. A household mail is still never retried (R-1361) | +| A signed job a box never took | the hub, on the 60 s sweep | `monitor.SignedJobWaitChecker`: a job still in the queue two hours after it was put there (a box asks every 15 minutes at the latest). The hub does not open a signed job, so it judges by age, not by the job's own end time | `signed_job_not_taken` (warning), once per job | it does not remove the job | +| A signed job a box took and refused, or whose run failed | the box's agent | the agent's clearing call (`DELETE …/jobs/`) now carries `outcome` (done · rejected · failed · malformed), a `reason` from a closed list (`signedjobs.ReasonCodes`) and the op's name in its query; before, „ran” and „refused” looked the same | `signed_job_refused` (error), once per job. The two clock reasons name the box's clock | an agent older than release 3 says nothing, and nothing is raised. The box's own words never reach a mail (unknown reason → „other”) | +| The guest-power watchdog gave up | the box's agent | the host report's `guest_power_gave_up` list (vmid, since, attempts); the watchdog's record is memory | `guest_power_gave_up` (error), once per down period; `guest_power_ok` (info) when a later report shows that guest running | „gone from the list” alone never clears (an agent restart empties the list while the guest is still down) | + +**Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate +and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red). + + ## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)] **The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic, diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index 1bc4a492..482958cd 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -374,6 +374,11 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by | `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` | | `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` | | `box_clock_wrong` | warning | **[built, not deployed — hub `main` after v0.181.0; needs an agent that stamps `sent_at`, built and not released]** the box's clock at the moment of sending is more than **one minute** from the hub's clock at arrival, in either direction (D79, `09` §3 decision 300); told once per direction. The hub cannot tell whose clock is wrong: a second box off in the same direction makes the alarm say „check the hub's own clock first”. An agent without `sent_at` is never judged | the first report within **30 seconds** → `box_clock_ok` (info) | `api/box_clock_test.go`; agent `hub/client_sentat_test.go` | +| `hub_daily_job_skipped` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: the newest slot of `deadline-check` or `offsite-key-audit` passed while the hub was not running; once per missed slot | none (the next slot runs) | `monitor/daily_ledger_test.go`, `cmd/hub/r1383_wiring_test.go` | +| `operator_mail_lost` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: operator mails of the last 48 h whose newest delivery record is `failed` — one mail for all, with each lost mail's own sentence; and a row (no mail) when three retries gave up | none | `notify/lost_operator_mail_test.go` | +| `signed_job_not_taken` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** a signed job is still in a box's queue after two hours; once per job | none (the job leaves the queue when the box takes it) | `monitor/signed_job_waits_test.go` | +| `signed_job_refused` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** needs an agent of release 3: the box cleared a signed job and said it was rejected, malformed, or its run failed; the cause is one of a closed list; once per job | none | `api/r1383_box_told_test.go`; agent `signedjobs/outcome_test.go` | +| `guest_power_gave_up` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** needs an agent of release 3: the host report lists a guest the guest-power watchdog stopped trying to start; once per down period | a later report shows that guest running → `guest_power_ok` (info) | `api/r1383_box_told_test.go`; agent `localapi/guestpower_test.go` | | `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` | | `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` | | `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` | diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt b/documentation/audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt new file mode 100644 index 00000000..61d07ede --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt @@ -0,0 +1,30 @@ +== MUTATION: the give-up is not listed (the journal line stays the only trace) + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s) + guestpower_test.go:271: give-up list = [], want guest 9201, 3 attempts, since 2026-11-14T02:30:00Z (the give-up's time, not a later tick's) +== MUTATION: a guest the watchdog still tries is listed as given up + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s) + guestpower_test.go:261: after 1 failed start(s) the watchdog still tries, but it reports a give-up: [{VMID:9201 Since:0001-01-01T00:00:00Z Attempts:1}] +== MUTATION: the give-up's time is not recorded + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s) + guestpower_test.go:271: give-up list = [{VMID:9201 Since:0001-01-01T00:00:00Z Attempts:3}], want guest 9201, 3 attempts, since 2026-11-14T02:30:00Z (the give-up's time, not a later tick's) +== MUTATION: an empty list is sent as a typed nil (the report would carry null) + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s) + guestpower_test.go:256: before any attempt the stanza is []localapi.GuestPowerGiveUp(nil), want an untyped nil (the report omits the field) + +4 mutations, 0 not a red-proof +== MUTATION: the outcome is not sent + RED on the mutant, GREEN on the real file + --- FAIL: TestCompleteJob_OutcomeRidesTheQuery (0.00s) + client_sentat_test.go:74: requests = ["/api/v1/hosts/demo-host-01/jobs/j1?" "/api/v1/hosts/demo-host-01/jobs/j2?"], want ["/api/v1/hosts/demo-host-01/jobs/j1?" "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&r + +1 mutations, 0 not a red-proof +== MUTATION: main does not wire the give-up into the report + RED on the mutant, GREEN on the real file + --- FAIL: TestMainWiresPVETrust (0.01s) + pvetrust_wiring_test.go:64: main.go wires "guest-power give-up → report" 0 time(s), want at least 1 (the two-call ones: at start and after a bundle install) + +1 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt b/documentation/audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt new file mode 100644 index 00000000..659c49da --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt @@ -0,0 +1,34 @@ +== MUTATION: a refused job is cleared with no word (the old behaviour) + RED on the mutant, GREEN on the real file + --- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s) + outcome_test.go:70: job "forged": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:malformed Op:storage_wipe} + outcome_test.go:70: job "expired": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:expired Op:storage_wipe} + outcome_test.go:70: job "early": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} +== MUTATION: a refused job is told as done + RED on the mutant, GREEN on the real file + --- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s) + outcome_test.go:70: job "early": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} + outcome_test.go:70: job "other-box": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:target Op:storage_wipe} + outcome_test.go:70: job "forged": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:malformed Op:storage_wipe} +== MUTATION: the runner never uses the outcome call + RED on the mutant, GREEN on the real file + --- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s) + outcome_test.go:70: job "early": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} + outcome_test.go:70: job "other-box": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:target Op:storage_wipe} + outcome_test.go:70: job "forged": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:malformed Op:storage_wipe} +== MUTATION: the two clock reasons are swapped + RED on the mutant, GREEN on the real file + --- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s) + outcome_test.go:70: job "expired": the hub was told {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} (told=true), want {Outcome:rejected Reason:expired Op:storage_wipe} + outcome_test.go:70: job "early": the hub was told {Outcome:rejected Reason:expired Op:storage_wipe} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} +== MUTATION: a failed run is told as done + RED on the mutant, GREEN on the real file + --- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s) + outcome_test.go:70: job "run-fails": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:failed Reason:execution_failed Op:storage_wipe} +== MUTATION: an unknown refusal sends the error's own text + RED on the mutant, GREEN on the real file + --- FAIL: TestReasonCode_ClosedList (0.00s) + outcome_test.go:92: reasonCode({Allowed:false Disposition: Reason:rejected Verified: Err:something new}) = "something new", not on ReasonCodes + outcome_test.go:96: an unknown refusal must be told as „other”, never as the error's own text + +6 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-hub-box-told.txt b/documentation/audits/night-build-2026-11-14/B/redproof-hub-box-told.txt new file mode 100644 index 00000000..f315e209 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-hub-box-told.txt @@ -0,0 +1,40 @@ +== MUTATION: a refused job is not told (executed or rejected look the same) + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s) + r1383_box_told_test.go:52: events=0, want ONE for the refused job +== MUTATION: a job that ran is told as refused + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.05s) + r1383_box_told_test.go:43: a job that ran, or an agent that says nothing, raised 2 event(s): [{ID:1 CustomerID:c1 EventType:signed_job_refused Severity:error Message:The signed job ran (agent_update) for h1 was REFUSED: the box ga +== MUTATION: a repeated DELETE is told twice + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s) + r1383_box_told_test.go:52: events=2, want ONE for the refused job +== MUTATION: the box's own reason text reaches the event + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s) + r1383_box_told_test.go:74: the failed run or the unknown words are not told as designed: + r1383_box_told_test.go:77: the box's own text reached an event: +== MUTATION: the op name is not checked + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s) + r1383_box_told_test.go:74: the failed run or the unknown words are not told as designed: + r1383_box_told_test.go:77: the box's own text reached an event: +== MUTATION: the give-up is told at every report + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s) + r1383_box_told_test.go:113: two reports of the same give-up: gave_up=2 ok=0, want ONE alarm +== MUTATION: the give-up is never told + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.04s) + r1383_box_told_test.go:113: two reports of the same give-up: gave_up=0 ok=0, want ONE alarm +== MUTATION: the entry vanishing clears the alarm (an agent restart reads as a recovery) + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s) + r1383_box_told_test.go:121: the entry vanished while the guest is still stopped: gave_up=1 ok=1 — that is not a recovery +== MUTATION: the alarm never clears + RED on the mutant, GREEN on the real file + --- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s) + r1383_box_told_test.go:130: after the guest runs again: gave_up=1 ok=0, want one recovery + +9 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-hub-daily-ledger.txt b/documentation/audits/night-build-2026-11-14/B/redproof-hub-daily-ledger.txt new file mode 100644 index 00000000..def929cf --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-hub-daily-ledger.txt @@ -0,0 +1,48 @@ +== MUTATION: the ledger never tells (the old behaviour: last run is forgotten at a restart) + RED on the mutant, GREEN on the real file + --- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s) + daily_ledger_test.go:59: events=0, want ONE for the slot the hub was down at: [] +== MUTATION: a slot that ran is told as skipped + RED on the mutant, GREEN on the real file + --- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s) + daily_ledger_test.go:54: a slot that ran was told as skipped: [|hub_daily_job_skipped|warning|hub|The hub's daily check „deadline-check” did NOT run at 2026-11-10 05:00 CET: the hub was not running at that minute. It last ran at 2 +== MUTATION: the same missed slot is told at every restart + RED on the mutant, GREEN on the real file + --- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s) + daily_ledger_test.go:73: the same missed slot was told 2 times +== MUTATION: the run is not recorded + RED on the mutant, GREEN on the real file + --- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s) + daily_ledger_test.go:54: a slot that ran was told as skipped: [|hub_daily_job_skipped|warning|hub|The hub's daily check „deadline-check” did NOT run at 2026-11-10 05:00 CET: the hub was not running at that minute. It last ran at 2 +== MUTATION: every job is watched + RED on the mutant, GREEN on the real file + --- FAIL: TestDailyLedger_UnwatchedJobAndNilLedger (0.00s) + daily_ledger_test.go:95: an unwatched job: events=[] state=map[daily_last:prune:2026-11-09 03:00:00 +0100 CET] +== MUTATION: the slot is taken as now minus 24 h, not a wall-clock time + RED on the mutant, GREEN on the real file + --- FAIL: TestPrevDailySlot_HourChangeDays (0.00s) + daily_ledger_test.go:113: PrevDailySlot(2026-10-25T04:30:00+01:00) = 2026-10-24T06:00:00+02:00, want 2026-10-24T05:00:00+02:00 + +6 mutations, 0 not a red-proof +== MUTATION: the scheduler does not ask the ledger at its start + RED on the mutant, GREEN on the real file + --- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s) + r1383_wiring_test.go:74: scheduleDaily calls the ledger: Started 0, Ran 1 — want 1 and 1 +== MUTATION: the scheduler does not record a run + RED on the mutant, GREEN on the real file + --- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s) + r1383_wiring_test.go:74: scheduleDaily calls the ledger: Started 1, Ran 0 — want 1 and 1 +== MUTATION: the lost-mail scan is never started + RED on the mutant, GREEN on the real file + --- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s) + r1383_wiring_test.go:70: main.go: lost-mail scan — found 0 time(s), want 1 +== MUTATION: the signed-job check is not on the sweep + RED on the mutant, GREEN on the real file + --- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s) + r1383_wiring_test.go:70: main.go: signed-job sweep — found 0 time(s), want 1 +== MUTATION: the ledger watches a job name nobody runs + RED on the mutant, GREEN on the real file + --- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s) + r1383_wiring_test.go:70: main.go: watched "offsite-key-audit" — found 0 time(s), want 1 + +5 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-hub-lost-mail.txt b/documentation/audits/night-build-2026-11-14/B/redproof-hub-lost-mail.txt new file mode 100644 index 00000000..3c1fb5ef --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-hub-lost-mail.txt @@ -0,0 +1,38 @@ +== MUTATION: the scan lists nothing + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s) + lost_operator_mail_test.go:70: lost mails told = 0, want 1 (the one whose retries the restart ate; not the one that went out on a retry) +== MUTATION: the scan's time is not kept — a second start tells the same mail + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s) + lost_operator_mail_test.go:89: a second start told 1 again (mails now 1) +== MUTATION: the summary is a row only, no mail + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s) + lost_operator_mail_test.go:73: mails sent at the start = [], want ONE to the operator +== MUTATION: the summary leaves no row + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s) + lost_operator_mail_test.go:85: event rows = [], want one operator_mail_lost row + +4 mutations, 0 not a red-proof +== MUTATION: a mail that went out on a retry counts as lost (every failed row) + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.05s) + lost_operator_mail_test.go:70: lost mails told = 2, want 1 (the one whose retries the restart ate; not the one that went out on a retry) +== MUTATION: a cooldown's suppressed row hides the failed mail + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s) + lost_operator_mail_test.go:70: lost mails told = 0, want 1 (the one whose retries the restart ate; not the one that went out on a retry) + +2 mutations, 0 not a red-proof +== MUTATION: the give-up leaves only the log line + RED on the mutant, GREEN on the real file + --- FAIL: TestLostOperatorMail_GiveUpLeavesARow (0.04s) + lost_operator_mail_test.go:106: after the retries gave up: event rows = [], want one that carries the lost mail's text +== MUTATION: the four cases' events reach the household + RED on the mutant, GREEN on the real file + --- FAIL: TestR1383_SilentFailureEventsAreOperatorOnly (0.00s) + lost_operator_mail_test.go:117: signed_job_refused must be operator-only + +2 mutations, 0 not a red-proof diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-hub-signed-job.txt b/documentation/audits/night-build-2026-11-14/B/redproof-hub-signed-job.txt new file mode 100644 index 00000000..a5f32545 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-hub-signed-job.txt @@ -0,0 +1,14 @@ +== MUTATION: a job that waits is never told + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s) + signed_job_waits_test.go:35: events=0, want ONE (the old job, once): [] +== MUTATION: a fresh job is told at once + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s) + signed_job_waits_test.go:35: events=2, want ONE (the old job, once): [cust-of-h1|signed_job_not_taken|warning|The signed job old for h1 is still in the queue after 3h0m0s: the box did not take it. A box asks for its jobs every 15 +== MUTATION: the job is told at every sweep + RED on the mutant, GREEN on the real file + --- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s) + signed_job_waits_test.go:35: events=2, want ONE (the old job, once): [cust-of-h1|signed_job_not_taken|warning|The signed job old for h1 is still in the queue after 3h0m0s: the box did not take it. A box asks for its jobs every 15 + +3 mutations, 0 not a red-proof diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 54017b71..11eea8e6 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -332,7 +332,7 @@ stopping line that lies. | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | | **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | -| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned