v0.83.0: observability pass — always-DEBUG capture ring + GET /debug/logs + heartbeat log-pull + gap-fill sweep

Capture layer: applog.New returns (logger, Ring) — slog fan-out, stderr at the
configured level, ~1000-entry ring fixed at LevelDebug (remote diagnostics
without a config flip). GET /debug/logs (token-authed, ?raw=1) + request-level
DEBUG middleware. Heartbeat log-pull mirrors the report logtail pattern:
envelope log_tail_requested -> next heartbeat carries log_tail (128KB cap,
consume-once, failed-push retry proven). Gap-fill sweep over netverify/
netstorage/netmount/signedjobs/selfupdate/disks/controller-swap/desired/loop.
Red-proofs: ring-at-emit-level FAILs capture test; drain removed FAILs
consume-once; dropped phase line FAILs the S7 log-sequence smoke.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-11 16:24:07 +02:00
parent 461eaf42c1
commit cb692f8788
20 changed files with 902 additions and 29 deletions
+33
View File
@@ -1,3 +1,36 @@
## v0.83.0 — observability pass: always-DEBUG capture ring + GET /debug/logs + heartbeat log-pull + gap-fill sweep (2026-07-11)
Agent half of the cross-repo observability task (controller v0.116.0 + hub v0.46.0). Motivating
incident: a refused NAS verify on an `info`-level box left NOTHING readable remotely — the agent
logged only to host journald and the new features emitted few lines.
- **Capture layer** (`internal/log`): `applog.New` now returns `(logger, *Ring)` — a slog fan-out
where stderr keeps the configured level (journald unchanged) and a ~1000-entry ring handler is
FIXED at `LevelDebug`, so flow detail exists for remote pulls without a config flip. The ring is
an io.Writer fed by a stdlib TextHandler; entries are parsed (time/level/message-verbatim).
Red-proof: ring gated at the emit level → capture-at-info test FAILS ("ring holds 1, want 2").
- **`GET /debug/logs`** (local API, same token-auth/self-scoping wrap as siblings): the ring as
JSON `{entries, total}`; `?raw=1` plain text; 503 when unwired. Plus a request-level DEBUG
middleware (method/path/status/duration — never bodies) wrapping the whole mux.
- **Heartbeat log-pull** (the report-channel logtail.go pattern mirrored): the control envelope
gains `log_tail_requested: bool` (additive); when set, the NEXT heartbeat carries
`log_tail: {collected_at, lines[]}` (newest-kept, 128 KB cap). Consume-once both ends: local
pending drains onto the carrying push; a FAILED push leaves the hub request pending → the next
envelope re-arms (retry proven in tests; red-proof: drain removed → tail ships every cycle →
FAIL). Serving a pull logs `operator log pull served` (INFO — customer-visible transparency).
- **Gap-fill sweep** (entry, decisions, outcome+duration, errors): netverify (job start, trigger
outcome, /proc/mounts verdict, journal byte-count, classification code, rollback outcome,
duration), netstorage add (pre-probe pass verdict, creds staged/removed — path only), netmount
Ensure/Remove (per-unit install/enable/remove-step results), signedjobs (jobs fetched ids+
duration, op received class/host/expiry — never signatures), selfupdate executor (invariants
passed, download sha-match+duration), disks (assign/eject/decommission outcome INFO),
ReassertGuestBinds (pass summary), controller-swap (pre-pull verify, negative health verdict),
desired syncer + hub loop (per-exchange DEBUG with durations).
- **S7 log-sequence smoke**: a full fake NAS add at emit level info must leave the ordered phase
markers in the ring (red-proof: dropping the /proc/mounts verdict line → FAIL naming the phase).
- No new sudoers grants, no journald scraping, no streaming — pull-only. Demo-deploy only — NOT
published (Peti stays 0.81.0; this reaches him with the next publish train).
## v0.82.0 — local-API version channel: X-Felhom-Agent-Version on every response (2026-07-11)
The controller's capability detection upgrades from route-probing to version comparison: the