docs: REPORT + CONTEXT for v0.111.0 (deployed + live-validated on 9201)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+16
-1
@@ -7,7 +7,22 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-07-09 (v0.106.0 — offsite provisioning SLICE 2: apply-bridge)
|
||||
Last updated: 2026-07-10 (v0.111.0 — remote app-log diagnostics; pairs with hub v0.43.0)
|
||||
|
||||
> **2026-07-10 — v0.111.0: remote app-log diagnostics (LIVE on 9201, pairs with hub v0.43.0).** The
|
||||
> telemetry scraper now attaches `LogIssue.Context` (±5 raw lines around the FIRST occurrence of each
|
||||
> error-severity issue; ≤11 lines, ≤400 chars/line, 16KB/report budget dropping lowest-count first; warns
|
||||
> carry none) and `metrics.RedactLine` sanitizes EVERY off-box context/tail line (password/token/api-key/
|
||||
> authorization/bearer → `[REDACTED]`, 64-hex → `[REDACTED-HEX64]`). On-demand log tails ride the ACK pull
|
||||
> pattern: hub ACK `log_tail_requests` → next report `log_tails` (200 lines via stacks.GetLogs /
|
||||
> FetchContainerLogTail, ordered, ≤64KB/app newest-kept, redacted, consume-once drain). Hub v0.43.0 stores
|
||||
> context (first-capture-wins + `context_customer` provenance), renders click-to-expand copyable issues,
|
||||
> fixes the period filter on Known Issues, replaces issue deletion with DISMISSAL (`dismissed_at`,
|
||||
> resurface only on `last_seen > dismissed_at`), adds `?customer=` filtered drill-down, and keeps the last
|
||||
> 2 tails per app with an ordered viewer + .log download. All red-proofs green (capture, redaction,
|
||||
> consume-once ×2, dismissal guard, range filter, context clobber). Live-proven on demo: synthetic error →
|
||||
> hub row with ordered 11-line context and `password=[REDACTED]`. OPERATOR: one click ("Request log tail"
|
||||
> on demo felhom-controller) completes the live tail round-trip — hub UI is password-gated, CC cannot.
|
||||
|
||||
> **2026-07-09 — v0.106.0: offsite provisioning SLICE 2 — the apply-bridge (pairs with hub v0.38.0).** On
|
||||
> startup the controller reconciles the hub-served `offsite:` descriptor into a key-only offbox target:
|
||||
|
||||
@@ -1,45 +1,69 @@
|
||||
# REPORT — controller v0.110.0: offbox stale-lock self-heal (C2) + crash-truthful status (C1) — DEPLOYED
|
||||
# REPORT — v0.111.0: remote app-log diagnostics (error context + on-demand log tails)
|
||||
|
||||
**Date:** 2026-07-10 · **Class:** implementation (controller only). Baseline v0.109.1 → **v0.110.0** (live
|
||||
on 9201, healthy). Fixes the overnight campaign's HIGH finding (C2) + the cosmetic C1. No hub/agent change.
|
||||
**Date:** 2026-07-10 · **Version:** v0.111.0 (from v0.110.0) · **Pairs with:** hub v0.43.0
|
||||
**Spec:** TASK — remote app-log diagnostics (Parts B, D-controller, E)
|
||||
|
||||
## What shipped
|
||||
- **C2 self-heal (`internal/backup`):** `resticStep` wraps the backup/prune/restore restic calls — on a lock
|
||||
error (`repository is already locked`) it escalates to `unlock --remove-all` and retries the step ONCE,
|
||||
safe because the repo has a SINGLE legitimate writer (per-customer sub-account isolation + the in-process
|
||||
single-flight mutex every caller holds). A second lock failure surfaces the error (never loops). Plus a
|
||||
cheap pre-run `unlock` (stale-only) hygiene before every run + restore. **Boundary (documented in code +
|
||||
README):** a DR-cloned second controller writing the same repo would defeat the single-writer premise —
|
||||
operator-supervised, out of scope.
|
||||
- **C1 (`NewManager.reconcileCrashedRun`):** a persisted `LastStatus="running"` at startup (a controller
|
||||
that died mid-run) flips to `error` + "megszakadt futás (a vezérlő újraindult futás közben)"; the next
|
||||
successful run clears it. Nil-settings guarded.
|
||||
|
||||
## Tests + companion red-proofs (ALL-GREEN, both red-proofs fired)
|
||||
- A — lock-on-backup → `unlock --remove-all` → retry → ok. **Red-proof:** neuter the escalation → the exact
|
||||
campaign failure `offbox backup rallly: exit status 1` → FAIL. Reverted.
|
||||
- B — lock persists twice → exactly one `--remove-all` + one retry → error surfaced, no unlock loop.
|
||||
- C — pre-run stale `unlock` issued on every run (asserted via the fake's call count).
|
||||
- E — no lock → `--remove-all` NEVER fires (the escalation can't trigger spuriously).
|
||||
- D (C1) — Manager built over `LastStatus="running"` flips to the Hungarian interrupted error. **Red-proof:**
|
||||
drop the flip → status lies "running" forever → FAIL. Reverted.
|
||||
### Part B — error context capture (`internal/metrics/logscanner.go`)
|
||||
- The scan loop was extracted into the pure `analyzeLogLines(lines []string)` (first-ever unit tests
|
||||
for the scanner). On the FIRST occurrence of an error-severity fingerprint in a scrape window,
|
||||
`captureContext` attaches `LogIssue.Context`: up to 5 raw lines before + the line + up to 5 after,
|
||||
ordered as emitted. Repeats never re-capture; warns never carry context.
|
||||
- Caps: ≤11 lines; each line ANSI-stripped + rune-truncated to 400 chars with `…`; and a HARD 16KB
|
||||
per-report budget in `internal/report/telemetry.go` (`enforceContextBudget` — context dropped from
|
||||
the lowest-count issues first, across all apps).
|
||||
- Additive `context` field (omitempty) on the report's issue objects — old hubs ignore it.
|
||||
|
||||
## Live validation (bounded, demo-safe — mirrors the campaign)
|
||||
- Deployed 0.110.0; demo untouched (escrowed, quota 50, ok).
|
||||
- **C1 live:** crashed the controller mid-run (`docker restart`) → on startup the log showed "previous run
|
||||
was interrupted by a controller restart…" and settings flipped to `last_status="error"`,
|
||||
`last_error="megszakadt futás…"`. ✓
|
||||
- **C2 live end-to-end:** crashed the controller DURING a backup (restic holding the exclusive lock) → the
|
||||
NEXT run self-healed and completed "backup OK: 2 app(s) backed up, 43s" with **zero manual restic**. The
|
||||
demo returned to `escrowed`/`ok`; the 04:15 schedule is safe. ✓
|
||||
- **Refinement vs the task premise:** the live evidence (and, on re-reading, the campaign's own teardown log
|
||||
where plain `restic unlock` already reported "successfully removed locks") shows the pre-run stale `unlock`
|
||||
hygiene ALONE clears these crash locks — the container's hostname is stable (`demo-felhom`), so restic
|
||||
detects the dead PID as stale. So in practice the lock is cleared before the backup step ever sees it; the
|
||||
`--remove-all` escalation is the genuine can't-detect-staleness fallback (unit-tested + red-proofed, not
|
||||
needed in this live run). Both layers ship — hygiene first, escalation as the safety net.
|
||||
### Part E — sanitization (`internal/metrics/redact.go`)
|
||||
- `RedactLine`: case-insensitive `password|passwd|secret|token|api[_-]?key|authorization|bearer`
|
||||
followed by `[=: ]\S+` → value `[REDACTED]` (the value group also swallows a `Bearer ` prefix so
|
||||
`Authorization: Bearer <tok>` redacts fully in one pass); any 64-hex string → `[REDACTED-HEX64]`.
|
||||
Applied to EVERY context line and tail line before it leaves the box — controller-side, authoritative.
|
||||
|
||||
## NEXT
|
||||
Peti onboarding runbook (the session-gated tracks: auto-confirm happy chain, fresh-enable full cycle, live
|
||||
quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end); old-box archive retirement (Track E
|
||||
hashes in hand — the new box restores byte-perfect).
|
||||
### Part D (controller half) — on-demand log tails (`internal/report/logtail.go`)
|
||||
- `PushResponse` gains `log_tail_requests: [app…]` (same additive ACK-flag pull pattern as
|
||||
escrow/config-refresh — the guest listens to no one). `OnPushResponse` → `SetPendingLogTails`;
|
||||
the NEXT `BuildReport` drains (consume-once) and ships `log_tails: [{app, collected_at, lines[]}]`.
|
||||
- Collection reuses existing plumbing: `stacks.GetLogs(app, 200)` (compose logs, ordered) for stacks;
|
||||
new `metrics.FetchContainerLogTail` (`docker logs --tail=200`, 15s timeout) for the
|
||||
`felhom-controller` container. Caps: ≤400 chars/line, ≤64KB/app HEAD-truncated (newest lines kept),
|
||||
every line redacted.
|
||||
- Fail-safe: fetch or push failure leaves the hub's request pending → the next ACK re-arms it.
|
||||
|
||||
## Tests + red-proofs (all executed; failures verbatim)
|
||||
|
||||
- **B red-proof** — capture call dropped → `TestAnalyzeLogLines_ErrorContextWindow` FAILED:
|
||||
`context has 0 lines, want 11: []string(nil)`. Restored → green.
|
||||
- **E red-proof** — `RedactLine` gutted (return s) → all redact tests FAILED with the secrets visible,
|
||||
e.g. `RedactLine("password=hunter2 token: abc Bearer xyz") = "password=hunter2 token: abc Bearer xyz"`
|
||||
and `64-hex secret shipped: "repo unlock with ab12…"`. Restored → green.
|
||||
- **D red-proof** — consume-once clear removed from `drainPendingLogTails` →
|
||||
`TestPendingLogTails_ConsumeOnce` FAILED: `second drain = [gokapi cwa] — tails would ship on EVERY
|
||||
report`. Restored → green.
|
||||
- Non-hollow coverage: exact ±5 ordered window; repeat keeps the FIRST occurrence's context; warn has
|
||||
no context; 400-char truncation (`…`); budget drops lowest-count first / under-budget untouched;
|
||||
window clamps at buffer edges; 10-issue cap + count ordering intact; tail order + redaction +
|
||||
line-cap + 64KB newest-kept + fetch-error skip + empty-ACK clears stale pending.
|
||||
- Full suite green (`go build && go vet && go test ./...`). One PRE-EXISTING flake observed once in
|
||||
`internal/web` (`TestBackupTier2Restore_DoubleClickRefused`, async timing) — passed 4/4 on re-run,
|
||||
unrelated to this diff.
|
||||
|
||||
## Deploy + live validation (demo 9201)
|
||||
|
||||
- Image `0.111.0` built on 180, deployed via the bootstrap mechanism →
|
||||
`gitea.dooplex.hu/admin/felhom-controller:0.111.0 Up (healthy)`, clean startup, channel up.
|
||||
- Synthetic error injected into the controller container's own log stream (the controller rides the
|
||||
same telemetry pipeline as apps): neighbor lines + `password=hunter2` + an `ERROR:` line —
|
||||
hub-side landing verified in the hub REPORT (felhom.eu) including LIVE redaction of the secret.
|
||||
- Method note: server-side pipeline exercised end-to-end (report cycle → hub ingest → hub DB); the
|
||||
hub UI click-path is render-tested (the hub operator password is Viktor-only — CC cannot log in).
|
||||
|
||||
## Residuals / notes
|
||||
|
||||
- The LIVE tail round-trip needs one operator click (hub → customer demo-felhom → App Telemetry →
|
||||
"Request log tail"); the tail then lands within ≤2 report cycles (≤30 min). The whole mechanism is
|
||||
unit- + API-test-proven on both repos including the consume-once red-proofs.
|
||||
- The live tail on 9201 would exercise the `FetchContainerLogTail` path (no customer app deployed on
|
||||
the demo right now); the `stacks.GetLogs` path is unit-covered via the fetch seam.
|
||||
- Do NOT request tails from Peti's box as part of this task — Viktor's call from the UI (per spec).
|
||||
|
||||
Reference in New Issue
Block a user