From c414ba53b56ec5ed803ee87540cfd18638a3d3eb Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 10 Jul 2026 16:25:37 +0200 Subject: [PATCH] docs: REPORT + CONTEXT for v0.111.0 (deployed + live-validated on 9201) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 17 ++++++++- REPORT.md | 102 +++++++++++++++++++++++++++++++++-------------------- 2 files changed, 79 insertions(+), 40 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index ff4e282..c1456a8 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,22 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-09 (v0.106.0 — offsite provisioning SLICE 2: apply-bridge) +Last updated: 2026-07-10 (v0.111.0 — remote app-log diagnostics; pairs with hub v0.43.0) + +> **2026-07-10 — v0.111.0: remote app-log diagnostics (LIVE on 9201, pairs with hub v0.43.0).** The +> telemetry scraper now attaches `LogIssue.Context` (±5 raw lines around the FIRST occurrence of each +> error-severity issue; ≤11 lines, ≤400 chars/line, 16KB/report budget dropping lowest-count first; warns +> carry none) and `metrics.RedactLine` sanitizes EVERY off-box context/tail line (password/token/api-key/ +> authorization/bearer → `[REDACTED]`, 64-hex → `[REDACTED-HEX64]`). On-demand log tails ride the ACK pull +> pattern: hub ACK `log_tail_requests` → next report `log_tails` (200 lines via stacks.GetLogs / +> FetchContainerLogTail, ordered, ≤64KB/app newest-kept, redacted, consume-once drain). Hub v0.43.0 stores +> context (first-capture-wins + `context_customer` provenance), renders click-to-expand copyable issues, +> fixes the period filter on Known Issues, replaces issue deletion with DISMISSAL (`dismissed_at`, +> resurface only on `last_seen > dismissed_at`), adds `?customer=` filtered drill-down, and keeps the last +> 2 tails per app with an ordered viewer + .log download. All red-proofs green (capture, redaction, +> consume-once ×2, dismissal guard, range filter, context clobber). Live-proven on demo: synthetic error → +> hub row with ordered 11-line context and `password=[REDACTED]`. OPERATOR: one click ("Request log tail" +> on demo felhom-controller) completes the live tail round-trip — hub UI is password-gated, CC cannot. > **2026-07-09 — v0.106.0: offsite provisioning SLICE 2 — the apply-bridge (pairs with hub v0.38.0).** On > startup the controller reconciles the hub-served `offsite:` descriptor into a key-only offbox target: diff --git a/REPORT.md b/REPORT.md index 6c26b5f..89f1623 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,45 +1,69 @@ -# REPORT — controller v0.110.0: offbox stale-lock self-heal (C2) + crash-truthful status (C1) — DEPLOYED +# REPORT — v0.111.0: remote app-log diagnostics (error context + on-demand log tails) -**Date:** 2026-07-10 · **Class:** implementation (controller only). Baseline v0.109.1 → **v0.110.0** (live -on 9201, healthy). Fixes the overnight campaign's HIGH finding (C2) + the cosmetic C1. No hub/agent change. +**Date:** 2026-07-10 · **Version:** v0.111.0 (from v0.110.0) · **Pairs with:** hub v0.43.0 +**Spec:** TASK — remote app-log diagnostics (Parts B, D-controller, E) ## What shipped -- **C2 self-heal (`internal/backup`):** `resticStep` wraps the backup/prune/restore restic calls — on a lock - error (`repository is already locked`) it escalates to `unlock --remove-all` and retries the step ONCE, - safe because the repo has a SINGLE legitimate writer (per-customer sub-account isolation + the in-process - single-flight mutex every caller holds). A second lock failure surfaces the error (never loops). Plus a - cheap pre-run `unlock` (stale-only) hygiene before every run + restore. **Boundary (documented in code + - README):** a DR-cloned second controller writing the same repo would defeat the single-writer premise — - operator-supervised, out of scope. -- **C1 (`NewManager.reconcileCrashedRun`):** a persisted `LastStatus="running"` at startup (a controller - that died mid-run) flips to `error` + "megszakadt futás (a vezérlő újraindult futás közben)"; the next - successful run clears it. Nil-settings guarded. -## Tests + companion red-proofs (ALL-GREEN, both red-proofs fired) -- A — lock-on-backup → `unlock --remove-all` → retry → ok. **Red-proof:** neuter the escalation → the exact - campaign failure `offbox backup rallly: exit status 1` → FAIL. Reverted. -- B — lock persists twice → exactly one `--remove-all` + one retry → error surfaced, no unlock loop. -- C — pre-run stale `unlock` issued on every run (asserted via the fake's call count). -- E — no lock → `--remove-all` NEVER fires (the escalation can't trigger spuriously). -- D (C1) — Manager built over `LastStatus="running"` flips to the Hungarian interrupted error. **Red-proof:** - drop the flip → status lies "running" forever → FAIL. Reverted. +### Part B — error context capture (`internal/metrics/logscanner.go`) +- The scan loop was extracted into the pure `analyzeLogLines(lines []string)` (first-ever unit tests + for the scanner). On the FIRST occurrence of an error-severity fingerprint in a scrape window, + `captureContext` attaches `LogIssue.Context`: up to 5 raw lines before + the line + up to 5 after, + ordered as emitted. Repeats never re-capture; warns never carry context. +- Caps: ≤11 lines; each line ANSI-stripped + rune-truncated to 400 chars with `…`; and a HARD 16KB + per-report budget in `internal/report/telemetry.go` (`enforceContextBudget` — context dropped from + the lowest-count issues first, across all apps). +- Additive `context` field (omitempty) on the report's issue objects — old hubs ignore it. -## Live validation (bounded, demo-safe — mirrors the campaign) -- Deployed 0.110.0; demo untouched (escrowed, quota 50, ok). -- **C1 live:** crashed the controller mid-run (`docker restart`) → on startup the log showed "previous run - was interrupted by a controller restart…" and settings flipped to `last_status="error"`, - `last_error="megszakadt futás…"`. ✓ -- **C2 live end-to-end:** crashed the controller DURING a backup (restic holding the exclusive lock) → the - NEXT run self-healed and completed "backup OK: 2 app(s) backed up, 43s" with **zero manual restic**. The - demo returned to `escrowed`/`ok`; the 04:15 schedule is safe. ✓ -- **Refinement vs the task premise:** the live evidence (and, on re-reading, the campaign's own teardown log - where plain `restic unlock` already reported "successfully removed locks") shows the pre-run stale `unlock` - hygiene ALONE clears these crash locks — the container's hostname is stable (`demo-felhom`), so restic - detects the dead PID as stale. So in practice the lock is cleared before the backup step ever sees it; the - `--remove-all` escalation is the genuine can't-detect-staleness fallback (unit-tested + red-proofed, not - needed in this live run). Both layers ship — hygiene first, escalation as the safety net. +### Part E — sanitization (`internal/metrics/redact.go`) +- `RedactLine`: case-insensitive `password|passwd|secret|token|api[_-]?key|authorization|bearer` + followed by `[=: ]\S+` → value `[REDACTED]` (the value group also swallows a `Bearer ` prefix so + `Authorization: Bearer ` redacts fully in one pass); any 64-hex string → `[REDACTED-HEX64]`. + Applied to EVERY context line and tail line before it leaves the box — controller-side, authoritative. -## NEXT -Peti onboarding runbook (the session-gated tracks: auto-confirm happy chain, fresh-enable full cycle, live -quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end); old-box archive retirement (Track E -hashes in hand — the new box restores byte-perfect). +### Part D (controller half) — on-demand log tails (`internal/report/logtail.go`) +- `PushResponse` gains `log_tail_requests: [app…]` (same additive ACK-flag pull pattern as + escrow/config-refresh — the guest listens to no one). `OnPushResponse` → `SetPendingLogTails`; + the NEXT `BuildReport` drains (consume-once) and ships `log_tails: [{app, collected_at, lines[]}]`. +- Collection reuses existing plumbing: `stacks.GetLogs(app, 200)` (compose logs, ordered) for stacks; + new `metrics.FetchContainerLogTail` (`docker logs --tail=200`, 15s timeout) for the + `felhom-controller` container. Caps: ≤400 chars/line, ≤64KB/app HEAD-truncated (newest lines kept), + every line redacted. +- Fail-safe: fetch or push failure leaves the hub's request pending → the next ACK re-arms it. + +## Tests + red-proofs (all executed; failures verbatim) + +- **B red-proof** — capture call dropped → `TestAnalyzeLogLines_ErrorContextWindow` FAILED: + `context has 0 lines, want 11: []string(nil)`. Restored → green. +- **E red-proof** — `RedactLine` gutted (return s) → all redact tests FAILED with the secrets visible, + e.g. `RedactLine("password=hunter2 token: abc Bearer xyz") = "password=hunter2 token: abc Bearer xyz"` + and `64-hex secret shipped: "repo unlock with ab12…"`. Restored → green. +- **D red-proof** — consume-once clear removed from `drainPendingLogTails` → + `TestPendingLogTails_ConsumeOnce` FAILED: `second drain = [gokapi cwa] — tails would ship on EVERY + report`. Restored → green. +- Non-hollow coverage: exact ±5 ordered window; repeat keeps the FIRST occurrence's context; warn has + no context; 400-char truncation (`…`); budget drops lowest-count first / under-budget untouched; + window clamps at buffer edges; 10-issue cap + count ordering intact; tail order + redaction + + line-cap + 64KB newest-kept + fetch-error skip + empty-ACK clears stale pending. +- Full suite green (`go build && go vet && go test ./...`). One PRE-EXISTING flake observed once in + `internal/web` (`TestBackupTier2Restore_DoubleClickRefused`, async timing) — passed 4/4 on re-run, + unrelated to this diff. + +## Deploy + live validation (demo 9201) + +- Image `0.111.0` built on 180, deployed via the bootstrap mechanism → + `gitea.dooplex.hu/admin/felhom-controller:0.111.0 Up (healthy)`, clean startup, channel up. +- Synthetic error injected into the controller container's own log stream (the controller rides the + same telemetry pipeline as apps): neighbor lines + `password=hunter2` + an `ERROR:` line — + hub-side landing verified in the hub REPORT (felhom.eu) including LIVE redaction of the secret. +- Method note: server-side pipeline exercised end-to-end (report cycle → hub ingest → hub DB); the + hub UI click-path is render-tested (the hub operator password is Viktor-only — CC cannot log in). + +## Residuals / notes + +- The LIVE tail round-trip needs one operator click (hub → customer demo-felhom → App Telemetry → + "Request log tail"); the tail then lands within ≤2 report cycles (≤30 min). The whole mechanism is + unit- + API-test-proven on both repos including the consume-once red-proofs. +- The live tail on 9201 would exercise the `FetchContainerLogTail` path (no customer app deployed on + the demo right now); the `stacks.GetLogs` path is unit-covered via the fetch seam. +- Do NOT request tails from Peti's box as part of this task — Viktor's call from the UI (per spec).