hub (unreleased): four silent failures tell the operator (R-1383, D78)
gates / gates (push) Successful in 2m9s

A hub daily check skipped while the hub was down; an operator mail that was never sent; a signed job a box did not
take; a signed job a box refused, and the guest-power watchdog's give-up (both from the agent). Operator-only.
No new table. Not deployed.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-14 17:29:53 +01:00
1 parent 88e636445a
commit 1886030782
22 files changed
+1269 -2

No files matched your search

@@ -645,6 +645,24 @@ never „now + 24 h", and the slot that ran last never runs again. On a 25-hour
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
2026-10-25 and 2027-03-28.
### 16.6 A job that does not run, gives up, or loses its state at a restart tells the operator [FACT — built, not released: hub `main` after v0.181.0 and agent `main` after 0.174.0, 2026-11-14; D78, `09` §3 decision 299, R-1383]
Four cases, the four the ruling names. Each leaves an event row and mails the operator only
(`notify.operatorOnlyEvents`). None needs a new table: the state is the hub's time-by-key settings
(`OSAlarmRaised`), the notification log and the signed-job queue.
| Case | Who sees it | How | Event | What it does NOT do |
|---|---|---|---|---|
| A hub daily check's slot passed while the hub was down | the hub, at its next start | `monitor.DailyLedger` keeps „the slot that last ran to its end” per watched job (`deadline-check`, `offsite-key-audit`) across a restart; `scheduleDaily` asks it at its start and tells it after each run | `hub_daily_job_skipped` (warning), once per missed slot | it does not run the missed check late. The snapshot and the prune are not watched (they make up for a missed slot by themselves). A job it has never seen is watched from the first start, not judged |
| An operator mail was never sent | the hub, at its next start | `Dispatcher.ReportLostOperatorMails`: at a start no retry timer is alive, so every operator mail of the last 48 hours whose newest delivery record is `failed` was never sent. ONE mail carries each lost mail's own sentence; the scan's time is kept, so a second restart does not repeat it | `operator_mail_lost` (error) | a mail that gave up while the hub keeps running leaves an event row at once and is MAILED only at the next start. The summary is a mail like any other and can be lost too; the next start lists it. A household mail is still never retried (R-1361) |
| A signed job a box never took | the hub, on the 60 s sweep | `monitor.SignedJobWaitChecker`: a job still in the queue two hours after it was put there (a box asks every 15 minutes at the latest). The hub does not open a signed job, so it judges by age, not by the job's own end time | `signed_job_not_taken` (warning), once per job | it does not remove the job |
| A signed job a box took and refused, or whose run failed | the box's agent | the agent's clearing call (`DELETE …/jobs/<id>`) now carries `outcome` (done · rejected · failed · malformed), a `reason` from a closed list (`signedjobs.ReasonCodes`) and the op's name in its query; before, „ran” and „refused” looked the same | `signed_job_refused` (error), once per job. The two clock reasons name the box's clock | an agent older than release 3 says nothing, and nothing is raised. The box's own words never reach a mail (unknown reason → „other”) |
| The guest-power watchdog gave up | the box's agent | the host report's `guest_power_gave_up` list (vmid, since, attempts); the watchdog's record is memory | `guest_power_gave_up` (error), once per down period; `guest_power_ok` (info) when a later report shows that guest running | „gone from the list” alone never clears (an agent restart empties the list while the guest is still down) |
**Not proven on a box or a deployed hub:** all five. The tests use fakes for the mail service and the real gate
and verifier for the signed jobs (`audits/night-build-2026-11-14/B/`, 43 mutations seen red).
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
@@ -374,6 +374,11 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
| `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` |
| `box_clock_wrong` | warning | **[built, not deployed — hub `main` after v0.181.0; needs an agent that stamps `sent_at`, built and not released]** the box's clock at the moment of sending is more than **one minute** from the hub's clock at arrival, in either direction (D79, `09` §3 decision 300); told once per direction. The hub cannot tell whose clock is wrong: a second box off in the same direction makes the alarm say „check the hub's own clock first”. An agent without `sent_at` is never judged | the first report within **30 seconds** → `box_clock_ok` (info) | `api/box_clock_test.go`; agent `hub/client_sentat_test.go` |
| `hub_daily_job_skipped` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: the newest slot of `deadline-check` or `offsite-key-audit` passed while the hub was not running; once per missed slot | none (the next slot runs) | `monitor/daily_ledger_test.go`, `cmd/hub/r1383_wiring_test.go` |
| `operator_mail_lost` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** at a hub start: operator mails of the last 48 h whose newest delivery record is `failed` — one mail for all, with each lost mail's own sentence; and a row (no mail) when three retries gave up | none | `notify/lost_operator_mail_test.go` |
| `signed_job_not_taken` | warning | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** a signed job is still in a box's queue after two hours; once per job | none (the job leaves the queue when the box takes it) | `monitor/signed_job_waits_test.go` |
| `signed_job_refused` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** needs an agent of release 3: the box cleared a signed job and said it was rejected, malformed, or its run failed; the cause is one of a closed list; once per job | none | `api/r1383_box_told_test.go`; agent `signedjobs/outcome_test.go` |
| `guest_power_gave_up` | error | **[built, not deployed — hub `main` after v0.181.0; D78, `05` §16.6]** needs an agent of release 3: the host report lists a guest the guest-power watchdog stopped trying to start; once per down period | a later report shows that guest running → `guest_power_ok` (info) | `api/r1383_box_told_test.go`; agent `localapi/guestpower_test.go` |
| `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` |
| `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` |
| `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` |
@@ -0,0 +1,30 @@
== MUTATION: the give-up is not listed (the journal line stays the only trace)
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
guestpower_test.go:271: give-up list = [], want guest 9201, 3 attempts, since 2026-11-14T02:30:00Z (the give-up's time, not a later tick's)
== MUTATION: a guest the watchdog still tries is listed as given up
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
guestpower_test.go:261: after 1 failed start(s) the watchdog still tries, but it reports a give-up: [{VMID:9201 Since:0001-01-01T00:00:00Z Attempts:1}]
== MUTATION: the give-up's time is not recorded
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
guestpower_test.go:271: give-up list = [{VMID:9201 Since:0001-01-01T00:00:00Z Attempts:3}], want guest 9201, 3 attempts, since 2026-11-14T02:30:00Z (the give-up's time, not a later tick's)
== MUTATION: an empty list is sent as a typed nil (the report would carry null)
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPower_GiveUpIsReportedUntilTheGuestRuns (0.00s)
guestpower_test.go:256: before any attempt the stanza is []localapi.GuestPowerGiveUp(nil), want an untyped nil (the report omits the field)
4 mutations, 0 not a red-proof
== MUTATION: the outcome is not sent
RED on the mutant, GREEN on the real file
--- FAIL: TestCompleteJob_OutcomeRidesTheQuery (0.00s)
client_sentat_test.go:74: requests = ["/api/v1/hosts/demo-host-01/jobs/j1?" "/api/v1/hosts/demo-host-01/jobs/j2?"], want ["/api/v1/hosts/demo-host-01/jobs/j1?" "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&r
1 mutations, 0 not a red-proof
== MUTATION: main does not wire the give-up into the report
RED on the mutant, GREEN on the real file
--- FAIL: TestMainWiresPVETrust (0.01s)
pvetrust_wiring_test.go:64: main.go wires "guest-power give-up → report" 0 time(s), want at least 1 (the two-call ones: at start and after a bundle install)
1 mutations, 0 not a red-proof
@@ -0,0 +1,34 @@
== MUTATION: a refused job is cleared with no word (the old behaviour)
RED on the mutant, GREEN on the real file
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
outcome_test.go:70: job "forged": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:malformed Op:storage_wipe}
outcome_test.go:70: job "expired": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:expired Op:storage_wipe}
outcome_test.go:70: job "early": the hub was told {Outcome: Reason: Op:} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
== MUTATION: a refused job is told as done
RED on the mutant, GREEN on the real file
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
outcome_test.go:70: job "early": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
outcome_test.go:70: job "other-box": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:target Op:storage_wipe}
outcome_test.go:70: job "forged": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:rejected Reason:malformed Op:storage_wipe}
== MUTATION: the runner never uses the outcome call
RED on the mutant, GREEN on the real file
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
outcome_test.go:70: job "early": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
outcome_test.go:70: job "other-box": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:target Op:storage_wipe}
outcome_test.go:70: job "forged": the hub was told {Outcome: Reason: Op:} (told=false), want {Outcome:rejected Reason:malformed Op:storage_wipe}
== MUTATION: the two clock reasons are swapped
RED on the mutant, GREEN on the real file
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
outcome_test.go:70: job "expired": the hub was told {Outcome:rejected Reason:not_yet_valid Op:storage_wipe} (told=true), want {Outcome:rejected Reason:expired Op:storage_wipe}
outcome_test.go:70: job "early": the hub was told {Outcome:rejected Reason:expired Op:storage_wipe} (told=true), want {Outcome:rejected Reason:not_yet_valid Op:storage_wipe}
== MUTATION: a failed run is told as done
RED on the mutant, GREEN on the real file
--- FAIL: TestRunner_TellsTheHubHowEachJobEnded (0.00s)
outcome_test.go:70: job "run-fails": the hub was told {Outcome:done Reason: Op:storage_wipe} (told=true), want {Outcome:failed Reason:execution_failed Op:storage_wipe}
== MUTATION: an unknown refusal sends the error's own text
RED on the mutant, GREEN on the real file
--- FAIL: TestReasonCode_ClosedList (0.00s)
outcome_test.go:92: reasonCode({Allowed:false Disposition: Reason:rejected Verified:<nil> Err:something new}) = "something new", not on ReasonCodes
outcome_test.go:96: an unknown refusal must be told as „other”, never as the error's own text
6 mutations, 0 not a red-proof
@@ -0,0 +1,40 @@
== MUTATION: a refused job is not told (executed or rejected look the same)
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
r1383_box_told_test.go:52: events=0, want ONE for the refused job
== MUTATION: a job that ran is told as refused
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.05s)
r1383_box_told_test.go:43: a job that ran, or an agent that says nothing, raised 2 event(s): [{ID:1 CustomerID:c1 EventType:signed_job_refused Severity:error Message:The signed job ran (agent_update) for h1 was REFUSED: the box ga
== MUTATION: a repeated DELETE is told twice
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
r1383_box_told_test.go:52: events=2, want ONE for the refused job
== MUTATION: the box's own reason text reaches the event
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
r1383_box_told_test.go:74: the failed run or the unknown words are not told as designed:
r1383_box_told_test.go:77: the box's own text reached an event:
== MUTATION: the op name is not checked
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobOutcome_RefusedIsToldDoneIsNot (0.04s)
r1383_box_told_test.go:74: the failed run or the unknown words are not told as designed:
r1383_box_told_test.go:77: the box's own text reached an event:
== MUTATION: the give-up is told at every report
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s)
r1383_box_told_test.go:113: two reports of the same give-up: gave_up=2 ok=0, want ONE alarm
== MUTATION: the give-up is never told
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.04s)
r1383_box_told_test.go:113: two reports of the same give-up: gave_up=0 ok=0, want ONE alarm
== MUTATION: the entry vanishing clears the alarm (an agent restart reads as a recovery)
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s)
r1383_box_told_test.go:121: the entry vanished while the guest is still stopped: gave_up=1 ok=1 — that is not a recovery
== MUTATION: the alarm never clears
RED on the mutant, GREEN on the real file
--- FAIL: TestGuestPowerGaveUp_ToldOnceClearedOnlyByARunningGuest (0.05s)
r1383_box_told_test.go:130: after the guest runs again: gave_up=1 ok=0, want one recovery
9 mutations, 0 not a red-proof
@@ -0,0 +1,48 @@
== MUTATION: the ledger never tells (the old behaviour: last run is forgotten at a restart)
RED on the mutant, GREEN on the real file
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
daily_ledger_test.go:59: events=0, want ONE for the slot the hub was down at: []
== MUTATION: a slot that ran is told as skipped
RED on the mutant, GREEN on the real file
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
daily_ledger_test.go:54: a slot that ran was told as skipped: [|hub_daily_job_skipped|warning|hub|The hub's daily check „deadline-check” did NOT run at 2026-11-10 05:00 CET: the hub was not running at that minute. It last ran at 2
== MUTATION: the same missed slot is told at every restart
RED on the mutant, GREEN on the real file
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
daily_ledger_test.go:73: the same missed slot was told 2 times
== MUTATION: the run is not recorded
RED on the mutant, GREEN on the real file
--- FAIL: TestDailyLedger_HubDownAtTheSlotIsTold (0.00s)
daily_ledger_test.go:54: a slot that ran was told as skipped: [|hub_daily_job_skipped|warning|hub|The hub's daily check „deadline-check” did NOT run at 2026-11-10 05:00 CET: the hub was not running at that minute. It last ran at 2
== MUTATION: every job is watched
RED on the mutant, GREEN on the real file
--- FAIL: TestDailyLedger_UnwatchedJobAndNilLedger (0.00s)
daily_ledger_test.go:95: an unwatched job: events=[] state=map[daily_last:prune:2026-11-09 03:00:00 +0100 CET]
== MUTATION: the slot is taken as now minus 24 h, not a wall-clock time
RED on the mutant, GREEN on the real file
--- FAIL: TestPrevDailySlot_HourChangeDays (0.00s)
daily_ledger_test.go:113: PrevDailySlot(2026-10-25T04:30:00+01:00) = 2026-10-24T06:00:00+02:00, want 2026-10-24T05:00:00+02:00
6 mutations, 0 not a red-proof
== MUTATION: the scheduler does not ask the ledger at its start
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
r1383_wiring_test.go:74: scheduleDaily calls the ledger: Started 0, Ran 1 — want 1 and 1
== MUTATION: the scheduler does not record a run
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
r1383_wiring_test.go:74: scheduleDaily calls the ledger: Started 1, Ran 0 — want 1 and 1
== MUTATION: the lost-mail scan is never started
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
r1383_wiring_test.go:70: main.go: lost-mail scan — found 0 time(s), want 1
== MUTATION: the signed-job check is not on the sweep
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
r1383_wiring_test.go:70: main.go: signed-job sweep — found 0 time(s), want 1
== MUTATION: the ledger watches a job name nobody runs
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_MainWiresTheSilentFailureChecks (0.00s)
r1383_wiring_test.go:70: main.go: watched "offsite-key-audit" — found 0 time(s), want 1
5 mutations, 0 not a red-proof
@@ -0,0 +1,38 @@
== MUTATION: the scan lists nothing
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
lost_operator_mail_test.go:70: lost mails told = 0, want 1 (the one whose retries the restart ate; not the one that went out on a retry)
== MUTATION: the scan's time is not kept — a second start tells the same mail
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
lost_operator_mail_test.go:89: a second start told 1 again (mails now 1)
== MUTATION: the summary is a row only, no mail
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
lost_operator_mail_test.go:73: mails sent at the start = [], want ONE to the operator
== MUTATION: the summary leaves no row
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
lost_operator_mail_test.go:85: event rows = [], want one operator_mail_lost row
4 mutations, 0 not a red-proof
== MUTATION: a mail that went out on a retry counts as lost (every failed row)
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.05s)
lost_operator_mail_test.go:70: lost mails told = 2, want 1 (the one whose retries the restart ate; not the one that went out on a retry)
== MUTATION: a cooldown's suppressed row hides the failed mail
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_ToldAtTheNextStart (0.04s)
lost_operator_mail_test.go:70: lost mails told = 0, want 1 (the one whose retries the restart ate; not the one that went out on a retry)
2 mutations, 0 not a red-proof
== MUTATION: the give-up leaves only the log line
RED on the mutant, GREEN on the real file
--- FAIL: TestLostOperatorMail_GiveUpLeavesARow (0.04s)
lost_operator_mail_test.go:106: after the retries gave up: event rows = [], want one that carries the lost mail's text
== MUTATION: the four cases' events reach the household
RED on the mutant, GREEN on the real file
--- FAIL: TestR1383_SilentFailureEventsAreOperatorOnly (0.00s)
lost_operator_mail_test.go:117: signed_job_refused must be operator-only
2 mutations, 0 not a red-proof
@@ -0,0 +1,14 @@
== MUTATION: a job that waits is never told
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s)
signed_job_waits_test.go:35: events=0, want ONE (the old job, once): []
== MUTATION: a fresh job is told at once
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s)
signed_job_waits_test.go:35: events=2, want ONE (the old job, once): [cust-of-h1|signed_job_not_taken|warning|The signed job old for h1 is still in the queue after 3h0m0s: the box did not take it. A box asks for its jobs every 15
== MUTATION: the job is told at every sweep
RED on the mutant, GREEN on the real file
--- FAIL: TestSignedJobWait_OldJobToldOnceFreshJobNot (0.00s)
signed_job_waits_test.go:35: events=2, want ONE (the old job, once): [cust-of-h1|signed_job_not_taken|warning|The signed job old for h1 is still in the queue after 3h0m0s: the box did not take it. A box asks for its jobs every 15
3 mutations, 0 not a red-proof
+1 -1
View File
@@ -332,7 +332,7 @@ stopping line that lies.
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |