From e35b1ae0e64e141602dfd883f17ae31c28f8a68b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 11 Jul 2026 16:57:47 +0200 Subject: [PATCH] =?UTF-8?q?hub=20v0.46.0:=20observability=20pass=20?= =?UTF-8?q?=E2=80=94=20per-box=20log=20pulls,=20bundle=20custody,=2072h=20?= =?UTF-8?q?TTL=20+=20secret=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit log_bundle_requests + log_bundles store (gzip, newest-3, 72h TTL purged on the 60s sweep); SaveLogBundle secret gate fail-closed (blocked flag row, no payload; REDACTED/checksums pass). Report ACK gains controller_log_requested + ingests controller_log_tail; heartbeat envelope gains log_tail_requested + ingests log_tail (consume-once on arrival; pre-0.83 agents stay visibly pending). Host detail Diagnostics section: request buttons (controller/agent), state rows with honest latency hints, View/Download endpoint. Red-proofs: gate disabled and clear-on-arrival removed both FAIL their tests. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- REUSE.md | 1 + hub/CHANGELOG.md | 28 ++ hub/cmd/hub/main.go | 6 + hub/internal/api/handler.go | 55 ++++ hub/internal/api/logbundle_ack_test.go | 119 +++++++++ hub/internal/store/logbundle.go | 271 ++++++++++++++++++++ hub/internal/store/logbundle_test.go | 147 +++++++++++ hub/internal/store/store.go | 28 ++ hub/internal/web/hosts.go | 3 + hub/internal/web/hosts_test.go | 14 +- hub/internal/web/logbundle.go | 154 +++++++++++ hub/internal/web/server.go | 16 +- hub/internal/web/templates/host_detail.html | 60 +++++ 13 files changed, 898 insertions(+), 4 deletions(-) create mode 100644 hub/internal/api/logbundle_ack_test.go create mode 100644 hub/internal/store/logbundle.go create mode 100644 hub/internal/store/logbundle_test.go create mode 100644 hub/internal/web/logbundle.go diff --git a/REUSE.md b/REUSE.md index 3a130cd..9695e1b 100644 --- a/REUSE.md +++ b/REUSE.md @@ -92,6 +92,7 @@ | `fileSHA256` | hub/internal/assets/assets.go (~L244) | `(path) (string, error)` | Streaming sha256 of a file | — | | `(*Store).SaveEvent` | hub/internal/store/store.go (~L1003) | `(...) (int64, error)` | Persisting ANY event (controller or hub source) | Pair with dispatcher/`onEvent` — saving alone never notifies. | | `(*Store).RequestLogTail` / `GetPendingLogTailRequests` / `SaveAppLogTail` | hub/internal/store/logtail.go | pending-intent + consume-once fulfillment | THE ACK-flag pull pattern for hub→box requests (copy for any new one) | SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box | +| `(*Store).RequestLogBundle` / `PendingLogBundleRequest` / `SaveLogBundle` / `PurgeExpiredLogBundles` | hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); `SaveLogBundle` runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; `[REDACTED]`/checksums pass by design | | `upsertAppIssue` dismissal/context semantics | hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on `excluded.last_seen > dismissed_at`; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) | | `store.GuestID` | hub/internal/store/store.go (~L1268) | `(hostID string, vmid int) string` | Canonical guest primary key | Never hand-concatenate host+vmid. | | `scheduleDaily` | hub/cmd/hub/main.go (~L449) | `(ctx, name, "HH:MM", fn, logger)` | Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. `parseHM` returns 0,0 (midnight) on bad input. | diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 88287de..a60cdd3 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,33 @@ # Felhom Hub — Changelog +## v0.46.0 — observability pass: per-box log pulls, bundle custody, TTL + secret gate (2026-07-11) + +Hub third of the cross-repo observability task (agent v0.83.0 + controller v0.116.0): remote, +pull-only access to both box components' always-DEBUG capture rings — honest to the sovereignty +posture (the hub never connects in; the box pushes on its own cycle and its own log records the +pull, customer-visible). + +- **Store** (`internal/store/logbundle.go` + schema): `log_bundle_requests` (one pending intent per + scope+component; scope = customer_id for the controller/report channel, host_id for the agent/ + heartbeat channel) + `log_bundles` (gzip payload, newest-3 retention, **72 h TTL** purged on the + existing 60 s sweep). `SaveLogBundle` runs the **token-pattern secret gate BEFORE storing** — + a hit stores a `blocked: possible secret` flag row with NO payload (fail-closed; WARN logged + hub-side); `[REDACTED]` shapes and public checksums deliberately pass (red-proof: gate disabled → + the planted `re_…` token stores → FAIL). +- **Channels** (additive, both directions): the report ACK gains `controller_log_requested` and + ingests `controller_log_tail`; the heartbeat envelope gains `log_tail_requested` and ingests + `log_tail`. Consume-once on arrival (red-proof: clear-on-arrival removed → the ACK re-advertises + forever → round-trip tests FAIL). A pre-0.83 agent simply never fulfills — the request stays + visibly `pending` (S6 tested), harmless. +- **UI** (host detail, English like the rest of the hub operator surface): a Diagnostics section + with **Request controller logs / Request agent logs** buttons (CSRF form posts), state rows + (`pending` with the honest per-channel latency hint — controller ≤ ~15 min report interval, + agent ≈ heartbeat cadence — / `available` with View+Download / `blocked`), and the 72 h custody + note. The hosts read-only invariant is amended: these two request forms are the ONLY actions + (pinned by test). +- Download endpoint `/hosts/{id}/log-bundles/{bid}[?download=1]` — session-authed like the rest of + the operator UI, scoped to the host's own channel scopes. + ## v0.45.0 — floor-UI separation + effective-floor source + per-box MinAgent conditional floor (2026-07-11) Two parts of the NAS/coupling backlog, both addressing the publish-train 0.81/0.113 floor footguns. diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index 65be4d9..9b09e48 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -448,6 +448,12 @@ func main() { hostMgmtPlaneChecker.Check() hostOOBChecker.Check() offsiteChecker.Check() + // v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL. + if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil { + logger.Printf("[WARN] log-bundle TTL purge failed: %v", perr) + } else if n > 0 { + logger.Printf("[INFO] log-bundle TTL purge: %d expired bundle(s) dropped", n) + } } } }() diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 0b02f3c..b30e1c4 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -331,6 +331,29 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) { } } + // Controller self-log tail (v0.46.0) — the controller's OWN debug ring, shipped on the + // cycle after the ACK's controller_log_requested. SaveLogBundle runs the secret gate + // (a hit stores a BLOCKED flag row, nothing else) and clears the pending request + // (consume-once). Backward-compatible: old controllers never send this. + var selfTailPayload struct { + ControllerLogTail *struct { + CollectedAt time.Time `json:"collected_at"` + Lines []string `json:"lines"` + } `json:"controller_log_tail"` + } + if err := json.Unmarshal(body, &selfTailPayload); err == nil && selfTailPayload.ControllerLogTail != nil { + lt := selfTailPayload.ControllerLogTail + blocked, berr := h.store.SaveLogBundle(payload.CustomerID, store.LogBundleComponentController, lt.CollectedAt, lt.Lines) + switch { + case berr != nil: + h.logger.Printf("[WARN] Failed to save controller log bundle for %s: %v", payload.CustomerID, berr) + case blocked: + h.logger.Printf("[WARN] controller log bundle for %s BLOCKED: possible secret in log content — nothing stored", payload.CustomerID) + default: + h.logger.Printf("[INFO] controller log bundle received for %s (%d lines)", payload.CustomerID, len(lt.Lines)) + } + } + // DR recipe — persist the controller's secret-free customer/apps half (preserving any host half). // Backward-compatible (old controllers won't have this field); a failure must not drop the report. var drPayload struct { @@ -376,6 +399,12 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) { resp["log_tail_requests"] = apps } + // v0.46.0 — pending CONTROLLER self-log pull (same additive ACK-flag pattern): the + // controller ships controller_log_tail on its NEXT report; omitted when nothing pending. + if pending, err := h.store.PendingLogBundleRequest(payload.CustomerID, store.LogBundleComponentController); err == nil && pending { + resp["controller_log_requested"] = true + } + // Phase 2 managed updates: advertise the effective controller-version FLOOR (per-customer override // else global default) and the latest available version. The controller compares its current // version against the floor and auto-updates when below it (latest stays the customer's opt-in @@ -447,6 +476,12 @@ type hostReportPayload struct { // DR recipe — the agent's storage/guest/PBS half (secret-free). RawMessage = stored verbatim, // ignore-unknown (forward-compat). Persisted to dr_recipe, assembled with the controller half. DRRecipe json.RawMessage `json:"dr_recipe"` + // LogTail (v0.46.0) — the agent's on-demand debug-ring tail, present only on the + // heartbeat right after the envelope's log_tail_requested (agent ≥ 0.83.0). + LogTail *struct { + CollectedAt time.Time `json:"collected_at"` + Lines []string `json:"lines"` + } `json:"log_tail"` } // drRecipeVersionOnly extracts just recipe_version from a half's JSON (ignore-unknown). 0 if absent. @@ -680,6 +715,21 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) { h.logger.Printf("[INFO] host-report from %s (%d guests, %d storage targets, %d backups, %d restore-tests, %d pbs-snapshots, %d bytes)", hostID, len(rep.Guests), len(rep.StorageTargets), len(rep.Backups), len(rep.RestoreTests), len(rep.PBSSnapshots), len(body)) + // Agent log tail (v0.46.0) — the debug-ring bundle a prior envelope requested. + // SaveLogBundle runs the secret gate + clears the pending request (consume-once). + // A failure must NOT drop the heartbeat; just warn. + if rep.LogTail != nil { + blocked, berr := h.store.SaveLogBundle(hostID, store.LogBundleComponentAgent, rep.LogTail.CollectedAt, rep.LogTail.Lines) + switch { + case berr != nil: + h.logger.Printf("[WARN] Failed to save agent log bundle for %s: %v", hostID, berr) + case blocked: + h.logger.Printf("[WARN] agent log bundle for %s BLOCKED: possible secret in log content — nothing stored", hostID) + default: + h.logger.Printf("[INFO] agent log bundle received for %s (%d lines)", hostID, len(rep.LogTail.Lines)) + } + } + // DR recipe — persist the agent's secret-free storage/guest/PBS half (preserving any app half). // A failure here must NOT drop the heartbeat (the report already saved); just warn. if len(rep.DRRecipe) > 0 && custID != "" { @@ -717,6 +767,11 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) { "desired_generation": desiredGen, "has_signed_ops": hasSignedOps, } + // v0.46.0 — pending agent log pull: the NEXT heartbeat carries log_tail (agent ≥ + // 0.83.0; older agents ignore the flag and the request stays visibly pending). + if pending, err := h.store.PendingLogBundleRequest(hostID, store.LogBundleComponentAgent); err == nil && pending { + resp["log_tail_requested"] = true + } w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode(resp) diff --git a/hub/internal/api/logbundle_ack_test.go b/hub/internal/api/logbundle_ack_test.go new file mode 100644 index 0000000..4fb1465 --- /dev/null +++ b/hub/internal/api/logbundle_ack_test.go @@ -0,0 +1,119 @@ +package api + +import ( + "encoding/json" + "net/http" + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// v0.46.0 (API half, controller channel) — the full pull round-trip on /report: +// 1. baseline ACK omits controller_log_requested +// 2. operator requests the controller bundle → the ACK advertises it +// 3. the next report carries controller_log_tail → stored + request cleared +// 4. the FOLLOWING ACK omits the flag again (consume-once) +func TestReportACK_ControllerLogRoundTrip(t *testing.T) { + h, st, _ := newTestHandler(t) + reportBody := `{"customer_id":"cust-a"}` + + rr := do(h, http.MethodPost, "/report", globalKey, reportBody) + if rr.Code != http.StatusOK { + t.Fatalf("report POST = %d (%s)", rr.Code, rr.Body.String()) + } + var ack map[string]json.RawMessage + json.Unmarshal(rr.Body.Bytes(), &ack) + if _, present := ack["controller_log_requested"]; present { + t.Fatalf("baseline ACK must omit controller_log_requested: %s", rr.Body.String()) + } + + if err := st.RequestLogBundle("cust-a", store.LogBundleComponentController); err != nil { + t.Fatal(err) + } + rr = do(h, http.MethodPost, "/report", globalKey, reportBody) + ack = map[string]json.RawMessage{} + json.Unmarshal(rr.Body.Bytes(), &ack) + if string(ack["controller_log_requested"]) != "true" { + t.Fatalf("ACK controller_log_requested = %s, want true", ack["controller_log_requested"]) + } + + tailReport := `{"customer_id":"cust-a","controller_log_tail":{"collected_at":"2026-07-11T10:00:00Z","lines":["[DEBUG] phase agent_add -> verifying","[INFO] operator log pull served (controller ring)"]}}` + rr = do(h, http.MethodPost, "/report", globalKey, tailReport) + if rr.Code != http.StatusOK { + t.Fatalf("tail report POST = %d", rr.Code) + } + bundles, err := st.GetLogBundles("cust-a") + if err != nil || len(bundles) != 1 || bundles[0].Component != store.LogBundleComponentController { + t.Fatalf("stored bundles = %+v err=%v, want 1 controller bundle", bundles, err) + } + _, lines, _ := st.GetLogBundleContent(bundles[0].ID, "cust-a") + if len(lines) != 2 || lines[0] != "[DEBUG] phase agent_add -> verifying" { + t.Fatalf("bundle lines not stored verbatim: %#v", lines) + } + + rr = do(h, http.MethodPost, "/report", globalKey, reportBody) + ack = map[string]json.RawMessage{} + json.Unmarshal(rr.Body.Bytes(), &ack) + if _, present := ack["controller_log_requested"]; present { + t.Fatalf("request survived fulfillment — the controller would ship every cycle: %s", rr.Body.String()) + } +} + +// v0.46.0 (API half, agent channel) — the heartbeat round-trip on /host-report, +// PLUS S6: a pre-0.83 agent that never ships log_tail leaves the request visibly +// PENDING across heartbeats (documented, harmless — the envelope just keeps +// advertising). Companion red-proof: clear the request at advertise time instead of +// arrival → the stays-pending assertion fails. +func TestHostReportEnvelope_AgentLogRoundTripAndOldAgentPending(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + + // Baseline: no flag. + rr := do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")) + var env map[string]json.RawMessage + json.Unmarshal(rr.Body.Bytes(), &env) + if _, present := env["log_tail_requested"]; present { + t.Fatalf("baseline envelope must omit log_tail_requested: %s", rr.Body.String()) + } + + // Request → advertised. + if err := st.RequestLogBundle("h1", store.LogBundleComponentAgent); err != nil { + t.Fatal(err) + } + rr = do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")) + env = map[string]json.RawMessage{} + json.Unmarshal(rr.Body.Bytes(), &env) + if string(env["log_tail_requested"]) != "true" { + t.Fatalf("envelope log_tail_requested = %s, want true", env["log_tail_requested"]) + } + + // S6: an old agent ignores the flag — after N plain heartbeats the request is STILL pending. + for i := 0; i < 3; i++ { + do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")) + } + if pending, _ := st.PendingLogBundleRequest("h1", store.LogBundleComponentAgent); !pending { + t.Fatal("pre-0.83 request vanished without fulfillment — it must stay visibly pending") + } + + // A 0.83 agent ships the tail → stored + cleared. + tailBody := `{"host_id":"h1","host":{},"guests":[],"storage_targets":[],"backups":[],"cloudflared":{},"audit_tail":[],` + + `"log_tail":{"collected_at":"2026-07-11T10:00:00Z","lines":["2026-07-11T10:00:00Z [INFO] operator log pull served"]}}` + rr = do(h, http.MethodPost, "/host-report", "HKEY", tailBody) + if rr.Code != http.StatusOK { + t.Fatalf("tail heartbeat = %d (%s)", rr.Code, rr.Body.String()) + } + bundles, err := st.GetLogBundles("h1") + if err != nil || len(bundles) != 1 || bundles[0].Component != store.LogBundleComponentAgent { + t.Fatalf("agent bundle = %+v err=%v", bundles, err) + } + if pending, _ := st.PendingLogBundleRequest("h1", store.LogBundleComponentAgent); pending { + t.Error("request survived fulfillment (consume-once broken)") + } + // The following envelope no longer advertises. + rr = do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")) + env = map[string]json.RawMessage{} + json.Unmarshal(rr.Body.Bytes(), &env) + if _, present := env["log_tail_requested"]; present { + t.Fatalf("envelope still advertises after fulfillment: %s", rr.Body.String()) + } +} diff --git a/hub/internal/store/logbundle.go b/hub/internal/store/logbundle.go new file mode 100644 index 0000000..ce945bd --- /dev/null +++ b/hub/internal/store/logbundle.go @@ -0,0 +1,271 @@ +package store + +import ( + "bytes" + "compress/gzip" + "database/sql" + "fmt" + "io" + "regexp" + "strings" + "time" +) + +// Component log bundles (v0.46.0 observability) — the box-component (controller / +// agent) sibling of the per-app log tails. Same sovereignty posture: the operator's +// request is a PENDING flag the box consumes on its own cycle (report ACK for the +// controller, heartbeat envelope for the agent); the hub never connects in. The +// received tail is gzip-stored with a 72 h TTL; the bundler runs a token-pattern +// gate BEFORE storing — a hit stores NOTHING and flags the bundle `blocked` (fail- +// closed belt over the "keys never values" logging convention). + +const ( + // LogBundleComponentController scopes by CUSTOMER (the report channel is per-customer). + LogBundleComponentController = "controller" + // LogBundleComponentAgent scopes by HOST (the heartbeat channel is per-host). + LogBundleComponentAgent = "agent" + + // logBundleTTL is how long a received bundle is served before the purge sweep drops it. + logBundleTTL = 72 * time.Hour + + // logBundleMaxRawBytes is the belt over the wire caps (both channels cap at 128 KiB): + // oversize input keeps the NEWEST lines within this budget. + logBundleMaxRawBytes = 1 << 20 + + // logBundleKeep bounds stored bundles per (scope, component) — newest kept. + logBundleKeep = 3 +) + +// secretPatterns is the bundle gate: content shaped like a credential. The logging +// conventions forbid secret VALUES in any log line, so a hit here is a violation — +// fail closed (store nothing, flag the bundle). Deliberately NOT a generic hex +// matcher: fingerprints/checksums are logged by design and are public. +var secretPatterns = []*regexp.Regexp{ + regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----`), + regexp.MustCompile(`\bre_[A-Za-z0-9]{10,}`), // Resend API key shape + regexp.MustCompile(`PVEAPIToken=[^\s"']+:\S+`), // Proxmox token WITH secret part + regexp.MustCompile(`(?i)\b(password|passwd|api_key|apikey|token_secret|client_secret)=[^\s"']{6,}`), + regexp.MustCompile(`Authorization:\s*Bearer\s+\S{16,}`), +} + +// findSecretPattern returns the name of the first matching pattern ("" = clean). +// A match containing "[REDACTED]" is skipped — box-side redaction deliberately +// leaves `password=[REDACTED]` shapes, which are proof of redaction, not a leak. Pure. +func findSecretPattern(lines []string) string { + for _, l := range lines { + for _, re := range secretPatterns { + if m := re.FindString(l); m != "" && !strings.Contains(m, "[REDACTED]") { + return re.String() + } + } + } + return "" +} + +// LogBundleRequest is one pending operator pull intent. +type LogBundleRequest struct { + ScopeID string + Component string + RequestedAt time.Time +} + +// LogBundleMeta is a stored bundle's row without the payload. +type LogBundleMeta struct { + ID int + ScopeID string + Component string + CollectedAt time.Time + ReceivedAt time.Time + SizeBytes int64 + Blocked bool + BlockedReason string +} + +// RequestLogBundle records (or refreshes) the operator's pending pull for one +// component. One active request per (scope, component) — a re-click refreshes. +func (s *Store) RequestLogBundle(scopeID, component string) error { + if component != LogBundleComponentController && component != LogBundleComponentAgent { + return fmt.Errorf("unknown log bundle component %q", component) + } + _, err := s.db.Exec(` + INSERT INTO log_bundle_requests (scope_id, component, requested_at) + VALUES (?, ?, ?) + ON CONFLICT(scope_id, component) DO UPDATE SET requested_at = excluded.requested_at`, + scopeID, component, time.Now().UTC()) + return err +} + +// PendingLogBundleRequest reports whether a pull is pending for (scope, component) +// — the value the report ACK / heartbeat envelope advertises. +func (s *Store) PendingLogBundleRequest(scopeID, component string) (bool, error) { + var n int + err := s.db.QueryRow(`SELECT COUNT(1) FROM log_bundle_requests WHERE scope_id = ? AND component = ?`, + scopeID, component).Scan(&n) + return n > 0, err +} + +// GetLogBundleRequests returns the pending requests for a scope (for the UI states). +func (s *Store) GetLogBundleRequests(scopeID string) ([]LogBundleRequest, error) { + rows, err := s.db.Query(`SELECT scope_id, component, requested_at FROM log_bundle_requests WHERE scope_id = ?`, scopeID) + if err != nil { + return nil, err + } + defer rows.Close() + var out []LogBundleRequest + for rows.Next() { + var r LogBundleRequest + if err := rows.Scan(&r.ScopeID, &r.Component, &r.RequestedAt); err == nil { + out = append(out, r) + } + } + return out, rows.Err() +} + +// SaveLogBundle stores a received component tail: secret-gate → (gzip+store | +// blocked row), prune to the newest logBundleKeep, and CLEAR the pending request +// (consume-once — same shape as SaveAppLogTail). Returns blocked=true when the +// gate fired (nothing stored beyond the flag row). +func (s *Store) SaveLogBundle(scopeID, component string, collectedAt time.Time, lines []string) (blocked bool, err error) { + lines = capBundleLines(lines, logBundleMaxRawBytes) + raw := strings.Join(lines, "\n") + + var gz []byte + reason := "" + if hit := findSecretPattern(lines); hit != "" { + // Fail-closed: a secret-shaped token means the content is NOT stored at all. + blocked = true + reason = "possible secret (pattern " + hit + ")" + } else { + var buf bytes.Buffer + zw := gzip.NewWriter(&buf) + if _, err := zw.Write([]byte(raw)); err != nil { + return false, err + } + if err := zw.Close(); err != nil { + return false, err + } + gz = buf.Bytes() + } + + tx, err := s.db.Begin() + if err != nil { + return blocked, err + } + defer tx.Rollback() + if _, err := tx.Exec(` + INSERT INTO log_bundles (scope_id, component, collected_at, received_at, size_bytes, gz, blocked, blocked_reason) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + scopeID, component, collectedAt, time.Now().UTC(), int64(len(raw)), gz, boolInt(blocked), reason); err != nil { + return blocked, err + } + if _, err := tx.Exec(` + DELETE FROM log_bundles + WHERE scope_id = ? AND component = ? AND id NOT IN ( + SELECT id FROM log_bundles WHERE scope_id = ? AND component = ? + ORDER BY id DESC LIMIT ? + )`, scopeID, component, scopeID, component, logBundleKeep); err != nil { + return blocked, err + } + // Consume-once: the fulfilled request is cleared (a blocked bundle also fulfills — + // re-requesting the same secret-carrying ring would just block again). + if _, err := tx.Exec(`DELETE FROM log_bundle_requests WHERE scope_id = ? AND component = ?`, + scopeID, component); err != nil { + return blocked, err + } + return blocked, tx.Commit() +} + +// GetLogBundles returns a scope's stored bundle rows (meta only), newest first. +func (s *Store) GetLogBundles(scopeID string) ([]LogBundleMeta, error) { + rows, err := s.db.Query(` + SELECT id, scope_id, component, collected_at, received_at, size_bytes, blocked, blocked_reason + FROM log_bundles WHERE scope_id = ? ORDER BY id DESC`, scopeID) + if err != nil { + return nil, err + } + defer rows.Close() + var out []LogBundleMeta + for rows.Next() { + m, serr := scanBundleMeta(rows) + if serr != nil { + continue + } + out = append(out, m) + } + return out, rows.Err() +} + +// GetLogBundleContent returns one bundle's meta + gunzipped lines, scoped (no +// cross-scope reads). A blocked bundle returns meta with nil lines. +func (s *Store) GetLogBundleContent(id int, scopeID string) (*LogBundleMeta, []string, error) { + row := s.db.QueryRow(` + SELECT id, scope_id, component, collected_at, received_at, size_bytes, blocked, blocked_reason, gz + FROM log_bundles WHERE id = ? AND scope_id = ?`, id, scopeID) + var m LogBundleMeta + var blockedInt int + var gz []byte + if err := row.Scan(&m.ID, &m.ScopeID, &m.Component, &m.CollectedAt, &m.ReceivedAt, + &m.SizeBytes, &blockedInt, &m.BlockedReason, &gz); err != nil { + if err == sql.ErrNoRows { + return nil, nil, nil + } + return nil, nil, err + } + m.Blocked = blockedInt != 0 + if m.Blocked || len(gz) == 0 { + return &m, nil, nil + } + zr, err := gzip.NewReader(bytes.NewReader(gz)) + if err != nil { + return &m, nil, err + } + raw, err := io.ReadAll(io.LimitReader(zr, logBundleMaxRawBytes+1)) + zr.Close() + if err != nil { + return &m, nil, err + } + return &m, strings.Split(string(raw), "\n"), nil +} + +// PurgeExpiredLogBundles drops bundles older than the 72 h TTL (the honest custody +// bound — pulled logs are transient diagnostics, not an archive). now is injectable +// for the S5 test. +func (s *Store) PurgeExpiredLogBundles(now time.Time) (int, error) { + res, err := s.db.Exec(`DELETE FROM log_bundles WHERE received_at < ?`, now.Add(-logBundleTTL).UTC()) + if err != nil { + return 0, err + } + n, _ := res.RowsAffected() + return int(n), nil +} + +func scanBundleMeta(rows *sql.Rows) (LogBundleMeta, error) { + var m LogBundleMeta + var blockedInt int + err := rows.Scan(&m.ID, &m.ScopeID, &m.Component, &m.CollectedAt, &m.ReceivedAt, + &m.SizeBytes, &blockedInt, &m.BlockedReason) + m.Blocked = blockedInt != 0 + return m, err +} + +// capBundleLines keeps the NEWEST lines within the raw byte budget (belt over the +// wire caps; mirrors the box-side newest-kept semantics). +func capBundleLines(lines []string, maxBytes int) []string { + total := 0 + start := len(lines) + for i := len(lines) - 1; i >= 0; i-- { + total += len(lines[i]) + 1 + if total > maxBytes { + break + } + start = i + } + return lines[start:] +} + +func boolInt(b bool) int { + if b { + return 1 + } + return 0 +} diff --git a/hub/internal/store/logbundle_test.go b/hub/internal/store/logbundle_test.go new file mode 100644 index 0000000..17032c5 --- /dev/null +++ b/hub/internal/store/logbundle_test.go @@ -0,0 +1,147 @@ +package store + +import ( + "io" + "log" + "path/filepath" + "strings" + "testing" + "time" +) + +func newBundleStore(t *testing.T) *Store { + t.Helper() + st, err := New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0)) + if err != nil { + t.Fatalf("store.New: %v", err) + } + t.Cleanup(func() { st.Close() }) + return st +} + +// Round-trip: request → pending → save → stored + request CLEARED (consume-once). +func TestLogBundle_RoundTripConsumeOnce(t *testing.T) { + st := newBundleStore(t) + if err := st.RequestLogBundle("host-1", LogBundleComponentAgent); err != nil { + t.Fatal(err) + } + if pending, _ := st.PendingLogBundleRequest("host-1", LogBundleComponentAgent); !pending { + t.Fatal("request not pending after RequestLogBundle") + } + blocked, err := st.SaveLogBundle("host-1", LogBundleComponentAgent, time.Now().UTC(), + []string{"2026-07-11T10:00:00Z [INFO] netmount: ensured network mount name=nas"}) + if err != nil || blocked { + t.Fatalf("save: blocked=%v err=%v", blocked, err) + } + if pending, _ := st.PendingLogBundleRequest("host-1", LogBundleComponentAgent); pending { + t.Error("request survived fulfillment — consume-once broken") + } + bundles, err := st.GetLogBundles("host-1") + if err != nil || len(bundles) != 1 || bundles[0].Blocked { + t.Fatalf("bundles = %+v err=%v, want 1 clean bundle", bundles, err) + } + meta, lines, err := st.GetLogBundleContent(bundles[0].ID, "host-1") + if err != nil || meta == nil || len(lines) != 1 || !strings.Contains(lines[0], "ensured network mount") { + t.Fatalf("content = %+v / %v / %v", meta, lines, err) + } + // Scoping: another scope cannot read it. + if m, _, _ := st.GetLogBundleContent(bundles[0].ID, "other-host"); m != nil { + t.Error("cross-scope bundle read must return nothing") + } +} + +// S4 secret gate: a planted Resend-shaped token → the bundle is BLOCKED, its content +// is NOT stored (nil lines, no gz), and the request is still cleared. Companion +// red-proof: empty the secretPatterns table → blocked=false + content stored → FAIL. +func TestLogBundle_SecretGateFailClosed(t *testing.T) { + st := newBundleStore(t) + if err := st.RequestLogBundle("cust-a", LogBundleComponentController); err != nil { + t.Fatal(err) + } + blocked, err := st.SaveLogBundle("cust-a", LogBundleComponentController, time.Now().UTC(), []string{ + "[INFO] all fine", + "[DEBUG] oops leaked key re_XZq81hd7wJq2M9Yv in a log line", + }) + if err != nil { + t.Fatal(err) + } + if !blocked { + t.Fatal("secret-shaped token not blocked — the gate is the last belt, it must fire") + } + bundles, _ := st.GetLogBundles("cust-a") + if len(bundles) != 1 || !bundles[0].Blocked || bundles[0].BlockedReason == "" { + t.Fatalf("blocked flag row missing/incomplete: %+v", bundles) + } + meta, lines, err := st.GetLogBundleContent(bundles[0].ID, "cust-a") + if err != nil || meta == nil || !meta.Blocked { + t.Fatalf("blocked meta = %+v err=%v", meta, err) + } + if lines != nil { + t.Fatalf("blocked bundle served content — nothing may be stored: %v", lines) + } + // The request is still consumed (re-pulling the same ring would block again). + if pending, _ := st.PendingLogBundleRequest("cust-a", LogBundleComponentController); pending { + t.Error("blocked save left the request pending") + } +} + +// The gate must NOT fire on deliberately-redacted values or public checksums. +func TestLogBundle_GateAllowsRedactedAndHashes(t *testing.T) { + st := newBundleStore(t) + blocked, err := st.SaveLogBundle("cust-a", LogBundleComponentController, time.Now().UTC(), []string{ + "[INFO] env applied password=[REDACTED]", + "[INFO] artifact sha256=9828c5f7aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaf50b", + "[DEBUG] leaf fingerprint_sha256=ab:cd:ef", + }) + if err != nil { + t.Fatal(err) + } + if blocked { + t.Fatal("gate false-positive on redacted/checksum content — legitimate diagnostics would never arrive") + } +} + +// S5 TTL: past 72 h → purged; before → served. Injectable clock. +func TestLogBundle_TTLPurge(t *testing.T) { + st := newBundleStore(t) + if _, err := st.SaveLogBundle("host-1", LogBundleComponentAgent, time.Now().UTC(), []string{"line"}); err != nil { + t.Fatal(err) + } + // Before the TTL: kept. + if n, err := st.PurgeExpiredLogBundles(time.Now().Add(71 * time.Hour)); err != nil || n != 0 { + t.Fatalf("purge at 71h dropped %d (err=%v), want 0", n, err) + } + if b, _ := st.GetLogBundles("host-1"); len(b) != 1 { + t.Fatal("bundle gone before the TTL") + } + // Past the TTL: dropped. + if n, err := st.PurgeExpiredLogBundles(time.Now().Add(73 * time.Hour)); err != nil || n != 1 { + t.Fatalf("purge at 73h dropped %d (err=%v), want 1", n, err) + } + if b, _ := st.GetLogBundles("host-1"); len(b) != 0 { + t.Fatal("expired bundle still served") + } +} + +// Retention: only the newest 3 bundles per (scope, component) are kept. +func TestLogBundle_KeepNewest(t *testing.T) { + st := newBundleStore(t) + for i := 0; i < 5; i++ { + if _, err := st.SaveLogBundle("host-1", LogBundleComponentAgent, time.Now().UTC(), []string{"line"}); err != nil { + t.Fatal(err) + } + } + b, _ := st.GetLogBundles("host-1") + if len(b) != 3 { + t.Fatalf("kept %d bundles, want 3 (newest)", len(b)) + } +} + +// Oversize input keeps the NEWEST lines within the raw budget (belt over wire caps). +func TestCapBundleLines_KeepsNewest(t *testing.T) { + lines := []string{"old " + strings.Repeat("x", 100), "new " + strings.Repeat("y", 100)} + capped := capBundleLines(lines, 110) + if len(capped) != 1 || !strings.HasPrefix(capped[0], "new") { + t.Fatalf("capped = %v, want only the newest line", capped) + } +} diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index d3b25f6..ae73c55 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -494,6 +494,34 @@ func (s *Store) migrate() error { return err } + // Component log bundles (v0.46.0 observability, see logbundle.go): the operator's + // pending pull intents per (scope, component) — scope = customer_id for the + // controller (report ACK channel) / host_id for the agent (heartbeat envelope) — + // plus the received gzip bundles (72 h TTL, purge on the 60 s sweep; a secret-gate + // hit stores a BLOCKED flag row with no payload). + _, err = s.db.Exec(` + CREATE TABLE IF NOT EXISTS log_bundle_requests ( + scope_id TEXT NOT NULL, + component TEXT NOT NULL, + requested_at DATETIME NOT NULL, + PRIMARY KEY (scope_id, component) + ); + CREATE TABLE IF NOT EXISTS log_bundles ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + scope_id TEXT NOT NULL, + component TEXT NOT NULL, + collected_at DATETIME NOT NULL, + received_at DATETIME NOT NULL, + size_bytes INTEGER NOT NULL, + gz BLOB, + blocked INTEGER NOT NULL DEFAULT 0, + blocked_reason TEXT NOT NULL DEFAULT '' + ); + `) + if err != nil { + return err + } + return nil } diff --git a/hub/internal/web/hosts.go b/hub/internal/web/hosts.go index 3c78a77..fdf7e8c 100644 --- a/hub/internal/web/hosts.go +++ b/hub/internal/web/hosts.go @@ -329,6 +329,9 @@ func (s *Server) handleHostDetail(w http.ResponseWriter, r *http.Request, hostID "StorageTargets": storageTargets, "DRPresent": drBundle != nil, "EscrowPresent": escrow != nil, + // v0.46.0 Diagnostics: pending log pulls + received/blocked bundles (72 h TTL). + "LogBundles": s.hostLogBundleRows(host), + "CSRFToken": s.getCSRFToken(r), } if err := s.templates.ExecuteTemplate(w, "host_detail.html", data); err != nil { s.logger.Printf("[ERROR] host_detail.html template: %v", err) diff --git a/hub/internal/web/hosts_test.go b/hub/internal/web/hosts_test.go index f7eafd0..f35dbf9 100644 --- a/hub/internal/web/hosts_test.go +++ b/hub/internal/web/hosts_test.go @@ -166,9 +166,17 @@ func TestHandleHostDetail(t *testing.T) { if strings.Contains(body, secretKey) { t.Errorf("SECRET LEAK: detail page rendered the host api_key") } - // Read-only: no host action buttons. - if strings.Contains(strings.ToLower(body), " 0 { + s.handleLogBundleView(w, r, rest[:i], rest[i+len("/log-bundles/"):]) + } else { + http.NotFound(w, r) + } case strings.HasPrefix(path, "/hosts/"): hostID := strings.TrimPrefix(path, "/hosts/") s.handleHostDetail(w, r, hostID) diff --git a/hub/internal/web/templates/host_detail.html b/hub/internal/web/templates/host_detail.html index b1c5be5..8ec0a3e 100644 --- a/hub/internal/web/templates/host_detail.html +++ b/hub/internal/web/templates/host_detail.html @@ -173,6 +173,66 @@ {{end}} + +
+

Diagnostics — Log Bundles

+

+ Pull-based: the box ships its debug ring on its own next cycle — controller ≤ one report interval (~15 min), + agent ≈ one heartbeat. The pull is recorded in the box's own log (customer-visible). Bundles expire after 72 h. +

+
+
+ + + +
+
+ + + +
+
+ {{if .LogBundles}} + + + + + + + + + + + + + {{range .LogBundles}} + + + + + + + + + {{end}} + +
ComponentStateCollectedReceivedSize
{{.Component}} + {{if eq .State "pending"}}pending + {{else if eq .State "blocked"}}blocked: possible secret + {{else}}available{{end}} + {{if .CollectedAt.IsZero}}—{{else}}{{timeAgo .CollectedAt}}{{end}}{{if .ReceivedAt.IsZero}}—{{else}}{{timeAgo .ReceivedAt}}{{end}}{{if .SizeBytes}}{{.SizeBytes}} B{{else}}—{{end}} + {{if eq .State "available"}} + View + Download + {{end}} +
+ {{else}} +
+

No log bundles. Use the request buttons above — the box delivers on its next cycle.

+
+ {{end}} +
+

DR / Backup