burn-down night watches: R-872 05:00 run worked as designed but silently (Tester 2 34.8 h old) — early returns now log why, check re-dated 2026-10-07; R-887: 2 lost jobs, both re-run green; session report

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 05:05:42 +02:00
parent c8774593f8
commit 307ecdf005
9 changed files with 144 additions and 3 deletions
@@ -112,3 +112,13 @@ cherry-picks onto `main`, writes CHANGELOG, closes rows, pushes and watches CI.
 | (agent, no row) | `go test ./internal/osupdate` red on DooPlex 21:25–01:55 (bundle test read the installer's new KEPT names as written files) — fixed | 10 | agent `b2b82ae` |
 | (register) | section headings recounted (they still carried the 199-row counts) | 5 | this batch |
 | R-516 | the bundle's last 49 formal forms → te-form; gate widened (61 → 0); narrowed to Go/template literals | 25 | controller `4c3c203` |
+
+## The two night watches (§5)
+
+| Watch | Result |
+|---|---|
+| R-872 at 05:00 | `Deadline check: 4 customers, 0 backup missed … 1 skipped (down)`. From a hub.db copy (+ -wal): Tester 2 first reported 2026-10-04 16:13:44Z → 34.8 h old at 05:00, under the 48 h line → not judged, by design, but SILENTLY. No alarm owed, none fired. Fixed without a row: the early returns now log why (hub, unreleased; red-proved). Dated check moved to 2026-10-07. `r872-watch.txt` |
+| R-887 lost CI jobs | **2** of ~30 runs: felhom.eu 1384 (21:23Z) and felhom-controller 1401 (00:45Z) — no log, failed after 10–13.5 min; each re-run once → success. `r887-lost-jobs.txt` |
+
+Seen, not filed (a documented design choice, F2 in `hub/internal/monitor/host_staleness.go`): a box already down when the
+hub restarts is announced again once — Tester 2 got a `host_down` operator mail at each of yesterday's six hub restarts.
@@ -0,0 +1,4 @@
+=== RUN   TestR872_DownButTooYoungIsLogged
+    r872_down_box_test.go:97: no 'not judged yet' line for a down box 34 h old; log:
+--- FAIL: TestR872_DownButTooYoungIsLogged (0.04s)
+FAIL
@@ -0,0 +1,7 @@
+== R-872 night watch, read 2026-10-06T03:02:06Z from a hub.db copy (+ -wal, -shm) — a different channel from the hub log
+host_reports True
+customer_configs True
+events True
+host_reports cols: ['id', 'host_id', 'customer_id', 'received_at', 'report_json', 'agent_version', 'cpu_percent', 'memory_percent']
+Tester-2: config status=('active',) host_reports=8 first host report=('2026-10-04 16:13:44',) last=('2026-10-04 18:05:48',)
+  recent events: [('host_down', '2026-10-05 22:50:08'), ('host_down', '2026-10-05 18:12:36'), ('host_down', '2026-10-05 13:26:03'), ('host_down', '2026-10-05 13:03:42'), ('host_down', '2026-10-05 12:31:37'), ('host_down', '2026-10-05 09:16:10')]
@@ -0,0 +1,4 @@
+1384 felhom.eu 4aa4d837 lost (no log, 21:23:23Z->21:33:38Z, 10m15s); rerun requested 21:34:51Z
+1384 rerun -> success (21:34:53Z->21:36:58Z, 2m05s)
+1401 felhom-controller c67b26be lost (no log, 00:45:01Z->00:58:38Z, 13m37s); rerun requested 00:59:06Z
+1401 rerun -> success (00:59:09Z->01:00:06Z, 57s)