gates / gates (push) Successful in 2m5s
- R-528: NOT closed. demo-felhom's night oom_check failed once (2026-11-11 04:40 UTC, Docker 29.9.3, containerd 2.4.2-3); 56 others passed; all four app_oom events since 10-12 explained. Due date 2026-11-28. This clears the due-checks gate that kept nightly CI red (runs 2628, 2630, 2632). - R-1391: Go 1.27.3 (hub, controller, agent) has GO-2026-6859; fixed in 1.27.4. - R-1392: the hub's "root files behind" compares the whole bundle sha, which moves with every agent release even when configs/ did not change; the os_config_bundle_behind alarm is due ~11-17 on all three boxes. - STATUS: D81 + the 11-14 do line; old Updated paragraph moved to STATUS-HISTORY. REPORT overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
43 lines
4.0 KiB
Markdown
43 lines
4.0 KiB
Markdown
# REPORT — health check after a few unattended days (2026-11-14, morning, attended)
|
||
|
||
**Plain words first.** The three test boxes are healthy. Backups ran last night on all three, off-site too, and each
|
||
restore proof passed. The night CI mail was red for three nights for one reason: a dated check (R-528) was due. The
|
||
check is now read and written down, and CI is green for it. Two new rows are filed. Nothing was changed on any box.
|
||
|
||
**Method.** Read only. The hub's own pages with CC's token (Dashboard, Hosts, System, Offsite); one copy of the hub
|
||
database **with its WAL** (newest host report 7 minutes old, `quick_check` ok; the copy was shredded after the read);
|
||
the hub log's deadline lines; the operator mailbox (label Monitoring, last 5 days); the CI run log of run id 2632
|
||
(job 2636) through the Gitea API.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Boxes | demo-felhom, demo-hp, Tester 1: controller 0.352.0, agent 0.173.0, ONLINE, all containers running, crash guard armed, 0 crash restarts |
|
||
| Backups last night | DB dumps 03:15–03:17 UTC; off-site `ok` 03:15–03:18 UTC; off-site restore proof `pass` 04:30 UTC (bentopdf, opengist, bookstack); whole-guest backups `success`; host restore tests `pass` (demo-hp from PBS 2026-11-14 02:43 UTC) |
|
||
| CI red ×3 nights (runs 2628, 2630, 2632) | only gate `due-checks`: R-528 was due 2026-11-12. Read and recorded (below), date moved to 2026-11-28 with the reason in the row |
|
||
| R-528 reading | NOT closed. One failed `oom_check` — demo-felhom 2026-11-11 04:40:30 UTC, Docker 29.9.3 / containerd 2.4.2-3: exit 137, `OOMKilled=false`, no `oom` event. 56 others passed. The four `app_oom` events since 2026-10-12 are all explained (a freeze, a squeeze test, a release proof) |
|
||
| Tester4 | Host deleted by the operator 07:54:31 UTC. The pairing-link mail at the same second is by design (a host delete that keeps the customer). The 05:00 `expected_dbdump_missed` came before the delete; the host's reports are gone with it, so the deadline check reads „not judged yet" from tomorrow (as Tester 2 does) |
|
||
| demo-hp blips | `agent_channel_unknown` 2026-11-11 13:52 → recovered 14:00 UTC (the certificate-pin spike on demo-hp that afternoon); `health_degraded` 2026-11-12 03:17 → recovered 03:22 UTC (a controller restart in the night step) |
|
||
| Security watch | New: 4 FileBrowser notices (one high, arbitrary file write) fixed only on 2.1.0 — belongs to R-1305; Go 1.27.3 GO-2026-6859 had no row → R-1391; cloudflared is R-1358 |
|
||
| Register | 248 rows at the start, 250 at the end. Opened R-1391 (Go 1.27.4), R-1392 (root files amber after every agent release). Closed 0. Dated line on R-528 |
|
||
|
||
## Findings
|
||
|
||
1. **R-1392 — the yellow „root files behind the vouched agent's" on all three boxes is a false amber, and it becomes
|
||
three operator alarms around 2026-11-17.** The files on the boxes are the vouched ones (`configs/` did not change
|
||
between 0.172.1 and 0.173.0), but the hub compares the whole bundle's sha256, which carries the version.
|
||
2. **R-1391 — Go 1.27.3 in hub, controller and agent; fixed in 1.27.4.** Reach not measured.
|
||
3. **CI turns red again after 2026-11-15:** R-1050 (HSTS one year) is due that day and needs a controller release.
|
||
|
||
## Waiting on the operator
|
||
|
||
- Rotate `GITEA_GIT_TOKEN` (the secret sweep mail of 2026-11-12 found it in 2 session records), then `sec put`.
|
||
- Approve the Proxmox set 9.2.26 and kernel 7.0.14-28 on the System page when you want Tester 1 to follow.
|
||
- R-1392: A or B (in the row). Sending the 0.173.0 bundle to the three boxes clears today's amber either way.
|
||
|
||
**Fixed without a row:** the local `instructions` gate (and its decoy test) failed because the memory index was 201
|
||
lines / 25 725 bytes (ceilings 200 / 25 600). Two index lines were merged; now 199 / 25 571. Not a repo file.
|
||
|
||
**Not done:** no box touched, no signed job sent, no hub change, no release. No area CHANGELOG entry: the change is
|
||
register-only (`hub/`, `scripts/`, `website/`, `marketing/` untouched). Provisioned nothing; teardown: the hub DB copy
|
||
in the scratchpad was shredded.
|