hub v0.46.0: observability pass — per-box log pulls, bundle custody, 72h TTL + secret gate

log_bundle_requests + log_bundles store (gzip, newest-3, 72h TTL purged on the
60s sweep); SaveLogBundle secret gate fail-closed (blocked flag row, no payload;
REDACTED/checksums pass). Report ACK gains controller_log_requested + ingests
controller_log_tail; heartbeat envelope gains log_tail_requested + ingests
log_tail (consume-once on arrival; pre-0.83 agents stay visibly pending). Host
detail Diagnostics section: request buttons (controller/agent), state rows with
honest latency hints, View/Download endpoint. Red-proofs: gate disabled and
clear-on-arrival removed both FAIL their tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-11 16:57:47 +02:00
parent 1ee5559772
commit e35b1ae0e6
13 changed files with 898 additions and 4 deletions
+1
View File
@@ -92,6 +92,7 @@
| `fileSHA256` | hub/internal/assets/assets.go (~L244) | `(path) (string, error)` | Streaming sha256 of a file | — |
| `(*Store).SaveEvent` | hub/internal/store/store.go (~L1003) | `(...) (int64, error)` | Persisting ANY event (controller or hub source) | Pair with dispatcher/`onEvent` — saving alone never notifies. |
| `(*Store).RequestLogTail` / `GetPendingLogTailRequests` / `SaveAppLogTail` | hub/internal/store/logtail.go | pending-intent + consume-once fulfillment | THE ACK-flag pull pattern for hub→box requests (copy for any new one) | SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box |
| `(*Store).RequestLogBundle` / `PendingLogBundleRequest` / `SaveLogBundle` / `PurgeExpiredLogBundles` | hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); `SaveLogBundle` runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; `[REDACTED]`/checksums pass by design |
| `upsertAppIssue` dismissal/context semantics | hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on `excluded.last_seen > dismissed_at`; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) |
| `store.GuestID` | hub/internal/store/store.go (~L1268) | `(hostID string, vmid int) string` | Canonical guest primary key | Never hand-concatenate host+vmid. |
| `scheduleDaily` | hub/cmd/hub/main.go (~L449) | `(ctx, name, "HH:MM", fn, logger)` | Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. `parseHM` returns 0,0 (midnight) on bad input. |
+28
View File
@@ -1,5 +1,33 @@
# Felhom Hub — Changelog
## v0.46.0 — observability pass: per-box log pulls, bundle custody, TTL + secret gate (2026-07-11)
Hub third of the cross-repo observability task (agent v0.83.0 + controller v0.116.0): remote,
pull-only access to both box components' always-DEBUG capture rings — honest to the sovereignty
posture (the hub never connects in; the box pushes on its own cycle and its own log records the
pull, customer-visible).
- **Store** (`internal/store/logbundle.go` + schema): `log_bundle_requests` (one pending intent per
scope+component; scope = customer_id for the controller/report channel, host_id for the agent/
heartbeat channel) + `log_bundles` (gzip payload, newest-3 retention, **72 h TTL** purged on the
existing 60 s sweep). `SaveLogBundle` runs the **token-pattern secret gate BEFORE storing**
a hit stores a `blocked: possible secret` flag row with NO payload (fail-closed; WARN logged
hub-side); `[REDACTED]` shapes and public checksums deliberately pass (red-proof: gate disabled →
the planted `re_…` token stores → FAIL).
- **Channels** (additive, both directions): the report ACK gains `controller_log_requested` and
ingests `controller_log_tail`; the heartbeat envelope gains `log_tail_requested` and ingests
`log_tail`. Consume-once on arrival (red-proof: clear-on-arrival removed → the ACK re-advertises
forever → round-trip tests FAIL). A pre-0.83 agent simply never fulfills — the request stays
visibly `pending` (S6 tested), harmless.
- **UI** (host detail, English like the rest of the hub operator surface): a Diagnostics section
with **Request controller logs / Request agent logs** buttons (CSRF form posts), state rows
(`pending` with the honest per-channel latency hint — controller ≤ ~15 min report interval,
agent ≈ heartbeat cadence — / `available` with View+Download / `blocked`), and the 72 h custody
note. The hosts read-only invariant is amended: these two request forms are the ONLY actions
(pinned by test).
- Download endpoint `/hosts/{id}/log-bundles/{bid}[?download=1]` — session-authed like the rest of
the operator UI, scoped to the host's own channel scopes.
## v0.45.0 — floor-UI separation + effective-floor source + per-box MinAgent conditional floor (2026-07-11)
Two parts of the NAS/coupling backlog, both addressing the publish-train 0.81/0.113 floor footguns.
+6
View File
@@ -448,6 +448,12 @@ func main() {
hostMgmtPlaneChecker.Check()
hostOOBChecker.Check()
offsiteChecker.Check()
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {
logger.Printf("[WARN] log-bundle TTL purge failed: %v", perr)
} else if n > 0 {
logger.Printf("[INFO] log-bundle TTL purge: %d expired bundle(s) dropped", n)
}
}
}
}()
+55
View File
@@ -331,6 +331,29 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) {
}
}
// Controller self-log tail (v0.46.0) — the controller's OWN debug ring, shipped on the
// cycle after the ACK's controller_log_requested. SaveLogBundle runs the secret gate
// (a hit stores a BLOCKED flag row, nothing else) and clears the pending request
// (consume-once). Backward-compatible: old controllers never send this.
var selfTailPayload struct {
ControllerLogTail *struct {
CollectedAt time.Time `json:"collected_at"`
Lines []string `json:"lines"`
} `json:"controller_log_tail"`
}
if err := json.Unmarshal(body, &selfTailPayload); err == nil && selfTailPayload.ControllerLogTail != nil {
lt := selfTailPayload.ControllerLogTail
blocked, berr := h.store.SaveLogBundle(payload.CustomerID, store.LogBundleComponentController, lt.CollectedAt, lt.Lines)
switch {
case berr != nil:
h.logger.Printf("[WARN] Failed to save controller log bundle for %s: %v", payload.CustomerID, berr)
case blocked:
h.logger.Printf("[WARN] controller log bundle for %s BLOCKED: possible secret in log content — nothing stored", payload.CustomerID)
default:
h.logger.Printf("[INFO] controller log bundle received for %s (%d lines)", payload.CustomerID, len(lt.Lines))
}
}
// DR recipe — persist the controller's secret-free customer/apps half (preserving any host half).
// Backward-compatible (old controllers won't have this field); a failure must not drop the report.
var drPayload struct {
@@ -376,6 +399,12 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) {
resp["log_tail_requests"] = apps
}
// v0.46.0 — pending CONTROLLER self-log pull (same additive ACK-flag pattern): the
// controller ships controller_log_tail on its NEXT report; omitted when nothing pending.
if pending, err := h.store.PendingLogBundleRequest(payload.CustomerID, store.LogBundleComponentController); err == nil && pending {
resp["controller_log_requested"] = true
}
// Phase 2 managed updates: advertise the effective controller-version FLOOR (per-customer override
// else global default) and the latest available version. The controller compares its current
// version against the floor and auto-updates when below it (latest stays the customer's opt-in
@@ -447,6 +476,12 @@ type hostReportPayload struct {
// DR recipe — the agent's storage/guest/PBS half (secret-free). RawMessage = stored verbatim,
// ignore-unknown (forward-compat). Persisted to dr_recipe, assembled with the controller half.
DRRecipe json.RawMessage `json:"dr_recipe"`
// LogTail (v0.46.0) — the agent's on-demand debug-ring tail, present only on the
// heartbeat right after the envelope's log_tail_requested (agent ≥ 0.83.0).
LogTail *struct {
CollectedAt time.Time `json:"collected_at"`
Lines []string `json:"lines"`
} `json:"log_tail"`
}
// drRecipeVersionOnly extracts just recipe_version from a half's JSON (ignore-unknown). 0 if absent.
@@ -680,6 +715,21 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
h.logger.Printf("[INFO] host-report from %s (%d guests, %d storage targets, %d backups, %d restore-tests, %d pbs-snapshots, %d bytes)",
hostID, len(rep.Guests), len(rep.StorageTargets), len(rep.Backups), len(rep.RestoreTests), len(rep.PBSSnapshots), len(body))
// Agent log tail (v0.46.0) — the debug-ring bundle a prior envelope requested.
// SaveLogBundle runs the secret gate + clears the pending request (consume-once).
// A failure must NOT drop the heartbeat; just warn.
if rep.LogTail != nil {
blocked, berr := h.store.SaveLogBundle(hostID, store.LogBundleComponentAgent, rep.LogTail.CollectedAt, rep.LogTail.Lines)
switch {
case berr != nil:
h.logger.Printf("[WARN] Failed to save agent log bundle for %s: %v", hostID, berr)
case blocked:
h.logger.Printf("[WARN] agent log bundle for %s BLOCKED: possible secret in log content — nothing stored", hostID)
default:
h.logger.Printf("[INFO] agent log bundle received for %s (%d lines)", hostID, len(rep.LogTail.Lines))
}
}
// DR recipe — persist the agent's secret-free storage/guest/PBS half (preserving any app half).
// A failure here must NOT drop the heartbeat (the report already saved); just warn.
if len(rep.DRRecipe) > 0 && custID != "" {
@@ -717,6 +767,11 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
"desired_generation": desiredGen,
"has_signed_ops": hasSignedOps,
}
// v0.46.0 — pending agent log pull: the NEXT heartbeat carries log_tail (agent ≥
// 0.83.0; older agents ignore the flag and the request stays visibly pending).
if pending, err := h.store.PendingLogBundleRequest(hostID, store.LogBundleComponentAgent); err == nil && pending {
resp["log_tail_requested"] = true
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(resp)
+119
View File
@@ -0,0 +1,119 @@
package api
import (
"encoding/json"
"net/http"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// v0.46.0 (API half, controller channel) — the full pull round-trip on /report:
// 1. baseline ACK omits controller_log_requested
// 2. operator requests the controller bundle → the ACK advertises it
// 3. the next report carries controller_log_tail → stored + request cleared
// 4. the FOLLOWING ACK omits the flag again (consume-once)
func TestReportACK_ControllerLogRoundTrip(t *testing.T) {
h, st, _ := newTestHandler(t)
reportBody := `{"customer_id":"cust-a"}`
rr := do(h, http.MethodPost, "/report", globalKey, reportBody)
if rr.Code != http.StatusOK {
t.Fatalf("report POST = %d (%s)", rr.Code, rr.Body.String())
}
var ack map[string]json.RawMessage
json.Unmarshal(rr.Body.Bytes(), &ack)
if _, present := ack["controller_log_requested"]; present {
t.Fatalf("baseline ACK must omit controller_log_requested: %s", rr.Body.String())
}
if err := st.RequestLogBundle("cust-a", store.LogBundleComponentController); err != nil {
t.Fatal(err)
}
rr = do(h, http.MethodPost, "/report", globalKey, reportBody)
ack = map[string]json.RawMessage{}
json.Unmarshal(rr.Body.Bytes(), &ack)
if string(ack["controller_log_requested"]) != "true" {
t.Fatalf("ACK controller_log_requested = %s, want true", ack["controller_log_requested"])
}
tailReport := `{"customer_id":"cust-a","controller_log_tail":{"collected_at":"2026-07-11T10:00:00Z","lines":["[DEBUG] phase agent_add -> verifying","[INFO] operator log pull served (controller ring)"]}}`
rr = do(h, http.MethodPost, "/report", globalKey, tailReport)
if rr.Code != http.StatusOK {
t.Fatalf("tail report POST = %d", rr.Code)
}
bundles, err := st.GetLogBundles("cust-a")
if err != nil || len(bundles) != 1 || bundles[0].Component != store.LogBundleComponentController {
t.Fatalf("stored bundles = %+v err=%v, want 1 controller bundle", bundles, err)
}
_, lines, _ := st.GetLogBundleContent(bundles[0].ID, "cust-a")
if len(lines) != 2 || lines[0] != "[DEBUG] phase agent_add -> verifying" {
t.Fatalf("bundle lines not stored verbatim: %#v", lines)
}
rr = do(h, http.MethodPost, "/report", globalKey, reportBody)
ack = map[string]json.RawMessage{}
json.Unmarshal(rr.Body.Bytes(), &ack)
if _, present := ack["controller_log_requested"]; present {
t.Fatalf("request survived fulfillment — the controller would ship every cycle: %s", rr.Body.String())
}
}
// v0.46.0 (API half, agent channel) — the heartbeat round-trip on /host-report,
// PLUS S6: a pre-0.83 agent that never ships log_tail leaves the request visibly
// PENDING across heartbeats (documented, harmless — the envelope just keeps
// advertising). Companion red-proof: clear the request at advertise time instead of
// arrival → the stays-pending assertion fails.
func TestHostReportEnvelope_AgentLogRoundTripAndOldAgentPending(t *testing.T) {
h, st, _ := newTestHandler(t)
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
// Baseline: no flag.
rr := do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1"))
var env map[string]json.RawMessage
json.Unmarshal(rr.Body.Bytes(), &env)
if _, present := env["log_tail_requested"]; present {
t.Fatalf("baseline envelope must omit log_tail_requested: %s", rr.Body.String())
}
// Request → advertised.
if err := st.RequestLogBundle("h1", store.LogBundleComponentAgent); err != nil {
t.Fatal(err)
}
rr = do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1"))
env = map[string]json.RawMessage{}
json.Unmarshal(rr.Body.Bytes(), &env)
if string(env["log_tail_requested"]) != "true" {
t.Fatalf("envelope log_tail_requested = %s, want true", env["log_tail_requested"])
}
// S6: an old agent ignores the flag — after N plain heartbeats the request is STILL pending.
for i := 0; i < 3; i++ {
do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1"))
}
if pending, _ := st.PendingLogBundleRequest("h1", store.LogBundleComponentAgent); !pending {
t.Fatal("pre-0.83 request vanished without fulfillment — it must stay visibly pending")
}
// A 0.83 agent ships the tail → stored + cleared.
tailBody := `{"host_id":"h1","host":{},"guests":[],"storage_targets":[],"backups":[],"cloudflared":{},"audit_tail":[],` +
`"log_tail":{"collected_at":"2026-07-11T10:00:00Z","lines":["2026-07-11T10:00:00Z [INFO] operator log pull served"]}}`
rr = do(h, http.MethodPost, "/host-report", "HKEY", tailBody)
if rr.Code != http.StatusOK {
t.Fatalf("tail heartbeat = %d (%s)", rr.Code, rr.Body.String())
}
bundles, err := st.GetLogBundles("h1")
if err != nil || len(bundles) != 1 || bundles[0].Component != store.LogBundleComponentAgent {
t.Fatalf("agent bundle = %+v err=%v", bundles, err)
}
if pending, _ := st.PendingLogBundleRequest("h1", store.LogBundleComponentAgent); pending {
t.Error("request survived fulfillment (consume-once broken)")
}
// The following envelope no longer advertises.
rr = do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1"))
env = map[string]json.RawMessage{}
json.Unmarshal(rr.Body.Bytes(), &env)
if _, present := env["log_tail_requested"]; present {
t.Fatalf("envelope still advertises after fulfillment: %s", rr.Body.String())
}
}
+271
View File
@@ -0,0 +1,271 @@
package store
import (
"bytes"
"compress/gzip"
"database/sql"
"fmt"
"io"
"regexp"
"strings"
"time"
)
// Component log bundles (v0.46.0 observability) — the box-component (controller /
// agent) sibling of the per-app log tails. Same sovereignty posture: the operator's
// request is a PENDING flag the box consumes on its own cycle (report ACK for the
// controller, heartbeat envelope for the agent); the hub never connects in. The
// received tail is gzip-stored with a 72 h TTL; the bundler runs a token-pattern
// gate BEFORE storing — a hit stores NOTHING and flags the bundle `blocked` (fail-
// closed belt over the "keys never values" logging convention).
const (
// LogBundleComponentController scopes by CUSTOMER (the report channel is per-customer).
LogBundleComponentController = "controller"
// LogBundleComponentAgent scopes by HOST (the heartbeat channel is per-host).
LogBundleComponentAgent = "agent"
// logBundleTTL is how long a received bundle is served before the purge sweep drops it.
logBundleTTL = 72 * time.Hour
// logBundleMaxRawBytes is the belt over the wire caps (both channels cap at 128 KiB):
// oversize input keeps the NEWEST lines within this budget.
logBundleMaxRawBytes = 1 << 20
// logBundleKeep bounds stored bundles per (scope, component) — newest kept.
logBundleKeep = 3
)
// secretPatterns is the bundle gate: content shaped like a credential. The logging
// conventions forbid secret VALUES in any log line, so a hit here is a violation —
// fail closed (store nothing, flag the bundle). Deliberately NOT a generic hex
// matcher: fingerprints/checksums are logged by design and are public.
var secretPatterns = []*regexp.Regexp{
regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----`),
regexp.MustCompile(`\bre_[A-Za-z0-9]{10,}`), // Resend API key shape
regexp.MustCompile(`PVEAPIToken=[^\s"']+:\S+`), // Proxmox token WITH secret part
regexp.MustCompile(`(?i)\b(password|passwd|api_key|apikey|token_secret|client_secret)=[^\s"']{6,}`),
regexp.MustCompile(`Authorization:\s*Bearer\s+\S{16,}`),
}
// findSecretPattern returns the name of the first matching pattern ("" = clean).
// A match containing "[REDACTED]" is skipped — box-side redaction deliberately
// leaves `password=[REDACTED]` shapes, which are proof of redaction, not a leak. Pure.
func findSecretPattern(lines []string) string {
for _, l := range lines {
for _, re := range secretPatterns {
if m := re.FindString(l); m != "" && !strings.Contains(m, "[REDACTED]") {
return re.String()
}
}
}
return ""
}
// LogBundleRequest is one pending operator pull intent.
type LogBundleRequest struct {
ScopeID string
Component string
RequestedAt time.Time
}
// LogBundleMeta is a stored bundle's row without the payload.
type LogBundleMeta struct {
ID int
ScopeID string
Component string
CollectedAt time.Time
ReceivedAt time.Time
SizeBytes int64
Blocked bool
BlockedReason string
}
// RequestLogBundle records (or refreshes) the operator's pending pull for one
// component. One active request per (scope, component) — a re-click refreshes.
func (s *Store) RequestLogBundle(scopeID, component string) error {
if component != LogBundleComponentController && component != LogBundleComponentAgent {
return fmt.Errorf("unknown log bundle component %q", component)
}
_, err := s.db.Exec(`
INSERT INTO log_bundle_requests (scope_id, component, requested_at)
VALUES (?, ?, ?)
ON CONFLICT(scope_id, component) DO UPDATE SET requested_at = excluded.requested_at`,
scopeID, component, time.Now().UTC())
return err
}
// PendingLogBundleRequest reports whether a pull is pending for (scope, component)
// — the value the report ACK / heartbeat envelope advertises.
func (s *Store) PendingLogBundleRequest(scopeID, component string) (bool, error) {
var n int
err := s.db.QueryRow(`SELECT COUNT(1) FROM log_bundle_requests WHERE scope_id = ? AND component = ?`,
scopeID, component).Scan(&n)
return n > 0, err
}
// GetLogBundleRequests returns the pending requests for a scope (for the UI states).
func (s *Store) GetLogBundleRequests(scopeID string) ([]LogBundleRequest, error) {
rows, err := s.db.Query(`SELECT scope_id, component, requested_at FROM log_bundle_requests WHERE scope_id = ?`, scopeID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []LogBundleRequest
for rows.Next() {
var r LogBundleRequest
if err := rows.Scan(&r.ScopeID, &r.Component, &r.RequestedAt); err == nil {
out = append(out, r)
}
}
return out, rows.Err()
}
// SaveLogBundle stores a received component tail: secret-gate → (gzip+store |
// blocked row), prune to the newest logBundleKeep, and CLEAR the pending request
// (consume-once — same shape as SaveAppLogTail). Returns blocked=true when the
// gate fired (nothing stored beyond the flag row).
func (s *Store) SaveLogBundle(scopeID, component string, collectedAt time.Time, lines []string) (blocked bool, err error) {
lines = capBundleLines(lines, logBundleMaxRawBytes)
raw := strings.Join(lines, "\n")
var gz []byte
reason := ""
if hit := findSecretPattern(lines); hit != "" {
// Fail-closed: a secret-shaped token means the content is NOT stored at all.
blocked = true
reason = "possible secret (pattern " + hit + ")"
} else {
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
if _, err := zw.Write([]byte(raw)); err != nil {
return false, err
}
if err := zw.Close(); err != nil {
return false, err
}
gz = buf.Bytes()
}
tx, err := s.db.Begin()
if err != nil {
return blocked, err
}
defer tx.Rollback()
if _, err := tx.Exec(`
INSERT INTO log_bundles (scope_id, component, collected_at, received_at, size_bytes, gz, blocked, blocked_reason)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
scopeID, component, collectedAt, time.Now().UTC(), int64(len(raw)), gz, boolInt(blocked), reason); err != nil {
return blocked, err
}
if _, err := tx.Exec(`
DELETE FROM log_bundles
WHERE scope_id = ? AND component = ? AND id NOT IN (
SELECT id FROM log_bundles WHERE scope_id = ? AND component = ?
ORDER BY id DESC LIMIT ?
)`, scopeID, component, scopeID, component, logBundleKeep); err != nil {
return blocked, err
}
// Consume-once: the fulfilled request is cleared (a blocked bundle also fulfills —
// re-requesting the same secret-carrying ring would just block again).
if _, err := tx.Exec(`DELETE FROM log_bundle_requests WHERE scope_id = ? AND component = ?`,
scopeID, component); err != nil {
return blocked, err
}
return blocked, tx.Commit()
}
// GetLogBundles returns a scope's stored bundle rows (meta only), newest first.
func (s *Store) GetLogBundles(scopeID string) ([]LogBundleMeta, error) {
rows, err := s.db.Query(`
SELECT id, scope_id, component, collected_at, received_at, size_bytes, blocked, blocked_reason
FROM log_bundles WHERE scope_id = ? ORDER BY id DESC`, scopeID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []LogBundleMeta
for rows.Next() {
m, serr := scanBundleMeta(rows)
if serr != nil {
continue
}
out = append(out, m)
}
return out, rows.Err()
}
// GetLogBundleContent returns one bundle's meta + gunzipped lines, scoped (no
// cross-scope reads). A blocked bundle returns meta with nil lines.
func (s *Store) GetLogBundleContent(id int, scopeID string) (*LogBundleMeta, []string, error) {
row := s.db.QueryRow(`
SELECT id, scope_id, component, collected_at, received_at, size_bytes, blocked, blocked_reason, gz
FROM log_bundles WHERE id = ? AND scope_id = ?`, id, scopeID)
var m LogBundleMeta
var blockedInt int
var gz []byte
if err := row.Scan(&m.ID, &m.ScopeID, &m.Component, &m.CollectedAt, &m.ReceivedAt,
&m.SizeBytes, &blockedInt, &m.BlockedReason, &gz); err != nil {
if err == sql.ErrNoRows {
return nil, nil, nil
}
return nil, nil, err
}
m.Blocked = blockedInt != 0
if m.Blocked || len(gz) == 0 {
return &m, nil, nil
}
zr, err := gzip.NewReader(bytes.NewReader(gz))
if err != nil {
return &m, nil, err
}
raw, err := io.ReadAll(io.LimitReader(zr, logBundleMaxRawBytes+1))
zr.Close()
if err != nil {
return &m, nil, err
}
return &m, strings.Split(string(raw), "\n"), nil
}
// PurgeExpiredLogBundles drops bundles older than the 72 h TTL (the honest custody
// bound — pulled logs are transient diagnostics, not an archive). now is injectable
// for the S5 test.
func (s *Store) PurgeExpiredLogBundles(now time.Time) (int, error) {
res, err := s.db.Exec(`DELETE FROM log_bundles WHERE received_at < ?`, now.Add(-logBundleTTL).UTC())
if err != nil {
return 0, err
}
n, _ := res.RowsAffected()
return int(n), nil
}
func scanBundleMeta(rows *sql.Rows) (LogBundleMeta, error) {
var m LogBundleMeta
var blockedInt int
err := rows.Scan(&m.ID, &m.ScopeID, &m.Component, &m.CollectedAt, &m.ReceivedAt,
&m.SizeBytes, &blockedInt, &m.BlockedReason)
m.Blocked = blockedInt != 0
return m, err
}
// capBundleLines keeps the NEWEST lines within the raw byte budget (belt over the
// wire caps; mirrors the box-side newest-kept semantics).
func capBundleLines(lines []string, maxBytes int) []string {
total := 0
start := len(lines)
for i := len(lines) - 1; i >= 0; i-- {
total += len(lines[i]) + 1
if total > maxBytes {
break
}
start = i
}
return lines[start:]
}
func boolInt(b bool) int {
if b {
return 1
}
return 0
}
+147
View File
@@ -0,0 +1,147 @@
package store
import (
"io"
"log"
"path/filepath"
"strings"
"testing"
"time"
)
func newBundleStore(t *testing.T) *Store {
t.Helper()
st, err := New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("store.New: %v", err)
}
t.Cleanup(func() { st.Close() })
return st
}
// Round-trip: request → pending → save → stored + request CLEARED (consume-once).
func TestLogBundle_RoundTripConsumeOnce(t *testing.T) {
st := newBundleStore(t)
if err := st.RequestLogBundle("host-1", LogBundleComponentAgent); err != nil {
t.Fatal(err)
}
if pending, _ := st.PendingLogBundleRequest("host-1", LogBundleComponentAgent); !pending {
t.Fatal("request not pending after RequestLogBundle")
}
blocked, err := st.SaveLogBundle("host-1", LogBundleComponentAgent, time.Now().UTC(),
[]string{"2026-07-11T10:00:00Z [INFO] netmount: ensured network mount name=nas"})
if err != nil || blocked {
t.Fatalf("save: blocked=%v err=%v", blocked, err)
}
if pending, _ := st.PendingLogBundleRequest("host-1", LogBundleComponentAgent); pending {
t.Error("request survived fulfillment — consume-once broken")
}
bundles, err := st.GetLogBundles("host-1")
if err != nil || len(bundles) != 1 || bundles[0].Blocked {
t.Fatalf("bundles = %+v err=%v, want 1 clean bundle", bundles, err)
}
meta, lines, err := st.GetLogBundleContent(bundles[0].ID, "host-1")
if err != nil || meta == nil || len(lines) != 1 || !strings.Contains(lines[0], "ensured network mount") {
t.Fatalf("content = %+v / %v / %v", meta, lines, err)
}
// Scoping: another scope cannot read it.
if m, _, _ := st.GetLogBundleContent(bundles[0].ID, "other-host"); m != nil {
t.Error("cross-scope bundle read must return nothing")
}
}
// S4 secret gate: a planted Resend-shaped token → the bundle is BLOCKED, its content
// is NOT stored (nil lines, no gz), and the request is still cleared. Companion
// red-proof: empty the secretPatterns table → blocked=false + content stored → FAIL.
func TestLogBundle_SecretGateFailClosed(t *testing.T) {
st := newBundleStore(t)
if err := st.RequestLogBundle("cust-a", LogBundleComponentController); err != nil {
t.Fatal(err)
}
blocked, err := st.SaveLogBundle("cust-a", LogBundleComponentController, time.Now().UTC(), []string{
"[INFO] all fine",
"[DEBUG] oops leaked key re_XZq81hd7wJq2M9Yv in a log line",
})
if err != nil {
t.Fatal(err)
}
if !blocked {
t.Fatal("secret-shaped token not blocked — the gate is the last belt, it must fire")
}
bundles, _ := st.GetLogBundles("cust-a")
if len(bundles) != 1 || !bundles[0].Blocked || bundles[0].BlockedReason == "" {
t.Fatalf("blocked flag row missing/incomplete: %+v", bundles)
}
meta, lines, err := st.GetLogBundleContent(bundles[0].ID, "cust-a")
if err != nil || meta == nil || !meta.Blocked {
t.Fatalf("blocked meta = %+v err=%v", meta, err)
}
if lines != nil {
t.Fatalf("blocked bundle served content — nothing may be stored: %v", lines)
}
// The request is still consumed (re-pulling the same ring would block again).
if pending, _ := st.PendingLogBundleRequest("cust-a", LogBundleComponentController); pending {
t.Error("blocked save left the request pending")
}
}
// The gate must NOT fire on deliberately-redacted values or public checksums.
func TestLogBundle_GateAllowsRedactedAndHashes(t *testing.T) {
st := newBundleStore(t)
blocked, err := st.SaveLogBundle("cust-a", LogBundleComponentController, time.Now().UTC(), []string{
"[INFO] env applied password=[REDACTED]",
"[INFO] artifact sha256=9828c5f7aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaf50b",
"[DEBUG] leaf fingerprint_sha256=ab:cd:ef",
})
if err != nil {
t.Fatal(err)
}
if blocked {
t.Fatal("gate false-positive on redacted/checksum content — legitimate diagnostics would never arrive")
}
}
// S5 TTL: past 72 h → purged; before → served. Injectable clock.
func TestLogBundle_TTLPurge(t *testing.T) {
st := newBundleStore(t)
if _, err := st.SaveLogBundle("host-1", LogBundleComponentAgent, time.Now().UTC(), []string{"line"}); err != nil {
t.Fatal(err)
}
// Before the TTL: kept.
if n, err := st.PurgeExpiredLogBundles(time.Now().Add(71 * time.Hour)); err != nil || n != 0 {
t.Fatalf("purge at 71h dropped %d (err=%v), want 0", n, err)
}
if b, _ := st.GetLogBundles("host-1"); len(b) != 1 {
t.Fatal("bundle gone before the TTL")
}
// Past the TTL: dropped.
if n, err := st.PurgeExpiredLogBundles(time.Now().Add(73 * time.Hour)); err != nil || n != 1 {
t.Fatalf("purge at 73h dropped %d (err=%v), want 1", n, err)
}
if b, _ := st.GetLogBundles("host-1"); len(b) != 0 {
t.Fatal("expired bundle still served")
}
}
// Retention: only the newest 3 bundles per (scope, component) are kept.
func TestLogBundle_KeepNewest(t *testing.T) {
st := newBundleStore(t)
for i := 0; i < 5; i++ {
if _, err := st.SaveLogBundle("host-1", LogBundleComponentAgent, time.Now().UTC(), []string{"line"}); err != nil {
t.Fatal(err)
}
}
b, _ := st.GetLogBundles("host-1")
if len(b) != 3 {
t.Fatalf("kept %d bundles, want 3 (newest)", len(b))
}
}
// Oversize input keeps the NEWEST lines within the raw budget (belt over wire caps).
func TestCapBundleLines_KeepsNewest(t *testing.T) {
lines := []string{"old " + strings.Repeat("x", 100), "new " + strings.Repeat("y", 100)}
capped := capBundleLines(lines, 110)
if len(capped) != 1 || !strings.HasPrefix(capped[0], "new") {
t.Fatalf("capped = %v, want only the newest line", capped)
}
}
+28
View File
@@ -494,6 +494,34 @@ func (s *Store) migrate() error {
return err
}
// Component log bundles (v0.46.0 observability, see logbundle.go): the operator's
// pending pull intents per (scope, component) — scope = customer_id for the
// controller (report ACK channel) / host_id for the agent (heartbeat envelope) —
// plus the received gzip bundles (72 h TTL, purge on the 60 s sweep; a secret-gate
// hit stores a BLOCKED flag row with no payload).
_, err = s.db.Exec(`
CREATE TABLE IF NOT EXISTS log_bundle_requests (
scope_id TEXT NOT NULL,
component TEXT NOT NULL,
requested_at DATETIME NOT NULL,
PRIMARY KEY (scope_id, component)
);
CREATE TABLE IF NOT EXISTS log_bundles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
scope_id TEXT NOT NULL,
component TEXT NOT NULL,
collected_at DATETIME NOT NULL,
received_at DATETIME NOT NULL,
size_bytes INTEGER NOT NULL,
gz BLOB,
blocked INTEGER NOT NULL DEFAULT 0,
blocked_reason TEXT NOT NULL DEFAULT ''
);
`)
if err != nil {
return err
}
return nil
}
+3
View File
@@ -329,6 +329,9 @@ func (s *Server) handleHostDetail(w http.ResponseWriter, r *http.Request, hostID
"StorageTargets": storageTargets,
"DRPresent": drBundle != nil,
"EscrowPresent": escrow != nil,
// v0.46.0 Diagnostics: pending log pulls + received/blocked bundles (72 h TTL).
"LogBundles": s.hostLogBundleRows(host),
"CSRFToken": s.getCSRFToken(r),
}
if err := s.templates.ExecuteTemplate(w, "host_detail.html", data); err != nil {
s.logger.Printf("[ERROR] host_detail.html template: %v", err)
+11 -3
View File
@@ -166,9 +166,17 @@ func TestHandleHostDetail(t *testing.T) {
if strings.Contains(body, secretKey) {
t.Errorf("SECRET LEAK: detail page rendered the host api_key")
}
// Read-only: no host action buttons.
if strings.Contains(strings.ToLower(body), "<button") {
t.Error("host detail must not contain action buttons")
// v0.46.0: the ONLY host actions are the two log-bundle request forms (the page is
// otherwise still read-only — no destructive/host-mutating buttons).
if got := strings.Count(strings.ToLower(body), "<button"); got != 2 {
t.Errorf("host detail has %d buttons, want exactly the 2 log-request buttons", got)
}
if strings.Count(body, `action="/hosts/demo-felhom-01/request-logs"`) != 2 {
t.Error("the request-logs forms are missing — every button must be a log-bundle request")
}
// The Diagnostics section renders with its honest latency hint.
if !strings.Contains(body, "Diagnostics") || !strings.Contains(body, "72 h") {
t.Error("Diagnostics log-bundle section missing")
}
}
+154
View File
@@ -0,0 +1,154 @@
package web
import (
"fmt"
"net/http"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// Component log-bundle UI (v0.46.0) — the host-detail "Diagnostics" section. The
// operator requests the CONTROLLER ring (rides the report ACK, ≤ ~15 min) or the
// AGENT ring (rides the heartbeat envelope, ≈ the heartbeat cadence); the box
// pushes on its own cycle (pull-only sovereignty posture; the box's own log
// records the pull — customer-visible transparency). Bundles expire after 72 h.
// logBundleRow is the template's per-bundle/pending row.
type logBundleRow struct {
ID int
Component string
State string // pending | available | blocked
RequestedAt time.Time
CollectedAt time.Time
ReceivedAt time.Time
SizeBytes int64
Blocked bool
BlockedNote string
}
// logBundleScope resolves the store scope for a component on this host: the agent
// channel is per-host; the controller channel is per-customer (the report ACK).
func logBundleScope(host *store.Host, component string) string {
if component == store.LogBundleComponentController {
return host.CustomerID
}
return host.HostID
}
// handleRequestLogBundle — POST /hosts/{id}/request-logs (form: component).
// Stores the pending pull the respective channel advertises on the box's next cycle.
func (s *Server) handleRequestLogBundle(w http.ResponseWriter, r *http.Request, hostID string) {
if !s.validateCSRF(r) {
http.Error(w, "Invalid CSRF token", http.StatusForbidden)
return
}
host, err := s.store.GetHost(hostID)
if err != nil || host == nil {
http.NotFound(w, r)
return
}
component := strings.TrimSpace(r.FormValue("component"))
if component != store.LogBundleComponentController && component != store.LogBundleComponentAgent {
http.Error(w, "Invalid component", http.StatusBadRequest)
return
}
if err := s.store.RequestLogBundle(logBundleScope(host, component), component); err != nil {
s.logger.Printf("[ERROR] RequestLogBundle %s/%s: %v", hostID, component, err)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
s.logger.Printf("[INFO] %s log bundle requested for host %s — the box delivers on its next cycle", component, hostID)
http.Redirect(w, r, "/hosts/"+hostID, http.StatusSeeOther)
}
// handleLogBundleView — GET /hosts/{id}/log-bundles/{bundleID} renders the bundle;
// ?download=1 serves it as a plain-text .log file. Scoped to the host's own scopes.
func (s *Server) handleLogBundleView(w http.ResponseWriter, r *http.Request, hostID, bundleIDStr string) {
host, err := s.store.GetHost(hostID)
if err != nil || host == nil {
http.NotFound(w, r)
return
}
id, err := strconv.Atoi(bundleIDStr)
if err != nil || id <= 0 {
http.NotFound(w, r)
return
}
// Try both scopes this host legitimately owns (agent = host_id, controller = customer_id).
meta, lines, err := s.store.GetLogBundleContent(id, host.HostID)
if err == nil && meta == nil {
meta, lines, err = s.store.GetLogBundleContent(id, host.CustomerID)
}
if err != nil {
s.logger.Printf("[ERROR] GetLogBundleContent %d/%s: %v", id, hostID, err)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
if meta == nil {
http.NotFound(w, r)
return
}
if meta.Blocked {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
fmt.Fprintf(w, "bundle %d (%s) is BLOCKED: %s\nNothing was stored — fix the offending log line box-side and re-request.\n",
meta.ID, meta.Component, meta.BlockedReason)
return
}
filename := fmt.Sprintf("%s-%s-%s.log", hostID, meta.Component, meta.CollectedAt.UTC().Format("20060102-150405"))
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
if r.URL.Query().Get("download") == "1" {
w.Header().Set("Content-Disposition", `attachment; filename="`+filename+`"`)
}
for _, line := range lines {
w.Write([]byte(line))
w.Write([]byte("\n"))
}
}
// hostLogBundleRows builds the host-detail Diagnostics rows: pending requests first
// (with the honest per-channel latency hint rendered template-side), then stored
// bundles newest-first. Both of this host's scopes are merged.
func (s *Server) hostLogBundleRows(host *store.Host) []logBundleRow {
var rows []logBundleRow
seenPending := map[string]bool{}
for _, scope := range []string{host.HostID, host.CustomerID} {
reqs, err := s.store.GetLogBundleRequests(scope)
if err != nil {
continue
}
for _, req := range reqs {
// A host row only owns its own channel scopes.
if logBundleScope(host, req.Component) != scope || seenPending[req.Component] {
continue
}
seenPending[req.Component] = true
rows = append(rows, logBundleRow{
Component: req.Component, State: "pending", RequestedAt: req.RequestedAt,
})
}
}
for _, scope := range []string{host.HostID, host.CustomerID} {
bundles, err := s.store.GetLogBundles(scope)
if err != nil {
continue
}
for _, b := range bundles {
if logBundleScope(host, b.Component) != scope {
continue
}
state := "available"
if b.Blocked {
state = "blocked"
}
rows = append(rows, logBundleRow{
ID: b.ID, Component: b.Component, State: state,
CollectedAt: b.CollectedAt, ReceivedAt: b.ReceivedAt,
SizeBytes: b.SizeBytes, Blocked: b.Blocked, BlockedNote: b.BlockedReason,
})
}
}
return rows
}
+15 -1
View File
@@ -251,9 +251,23 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Offsite — read-only WG endpoint + peer registry (S2). Mutations stay on the admin API.
case path == "/offsite":
s.handleOffsite(w, r)
// Hosts — read-only fleet view (audit F-M1). GET only; no host actions.
// Hosts — read-only fleet view (audit F-M1) + the v0.46.0 log-bundle actions.
case path == "/hosts" || path == "/hosts/":
s.handleHostsList(w, r)
case strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/request-logs"):
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/request-logs")
if r.Method == http.MethodPost {
s.handleRequestLogBundle(w, r, hostID)
} else {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
case strings.HasPrefix(path, "/hosts/") && strings.Contains(path, "/log-bundles/"):
rest := strings.TrimPrefix(path, "/hosts/")
if i := strings.Index(rest, "/log-bundles/"); i > 0 {
s.handleLogBundleView(w, r, rest[:i], rest[i+len("/log-bundles/"):])
} else {
http.NotFound(w, r)
}
case strings.HasPrefix(path, "/hosts/"):
hostID := strings.TrimPrefix(path, "/hosts/")
s.handleHostDetail(w, r, hostID)
@@ -173,6 +173,66 @@
{{end}}
</section>
<!-- Diagnostics: component log bundles (v0.46.0) -->
<section class="card">
<h2>Diagnostics — Log Bundles</h2>
<p class="hint" style="color: var(--text-muted); font-size: 0.85rem;">
Pull-based: the box ships its debug ring on its own next cycle — controller &le; one report interval (~15 min),
agent &asymp; one heartbeat. The pull is recorded in the box's own log (customer-visible). Bundles expire after 72 h.
</p>
<div style="display: flex; gap: 0.5rem; margin: 0.75rem 0;">
<form method="POST" action="/hosts/{{.HostID}}/request-logs" style="display: inline;">
<input type="hidden" name="_csrf" value="{{.CSRFToken}}">
<input type="hidden" name="component" value="controller">
<button type="submit" class="btn btn-sm">Request controller logs</button>
</form>
<form method="POST" action="/hosts/{{.HostID}}/request-logs" style="display: inline;">
<input type="hidden" name="_csrf" value="{{.CSRFToken}}">
<input type="hidden" name="component" value="agent">
<button type="submit" class="btn btn-sm">Request agent logs</button>
</form>
</div>
{{if .LogBundles}}
<table class="data-table">
<thead>
<tr>
<th>Component</th>
<th>State</th>
<th>Collected</th>
<th>Received</th>
<th>Size</th>
<th></th>
</tr>
</thead>
<tbody>
{{range .LogBundles}}
<tr>
<td>{{.Component}}</td>
<td>
{{if eq .State "pending"}}<span class="badge badge-neutral" title="Waiting for the box's next cycle (requested {{timeAgo .RequestedAt}})">pending</span>
{{else if eq .State "blocked"}}<span class="badge badge-error" title="{{.BlockedNote}}">blocked: possible secret</span>
{{else}}<span class="badge badge-ok">available</span>{{end}}
</td>
<td>{{if .CollectedAt.IsZero}}—{{else}}{{timeAgo .CollectedAt}}{{end}}</td>
<td>{{if .ReceivedAt.IsZero}}—{{else}}{{timeAgo .ReceivedAt}}{{end}}</td>
<td>{{if .SizeBytes}}{{.SizeBytes}} B{{else}}—{{end}}</td>
<td>
{{if eq .State "available"}}
<a href="/hosts/{{$.HostID}}/log-bundles/{{.ID}}" class="btn btn-sm">View</a>
<a href="/hosts/{{$.HostID}}/log-bundles/{{.ID}}?download=1" class="btn btn-sm">Download</a>
{{end}}
</td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<div class="empty-state" style="border: none;">
<p>No log bundles. Use the request buttons above — the box delivers on its next cycle.</p>
</div>
{{end}}
</section>
<!-- DR / Backup -->
<section class="card">
<h2>DR / Backup</h2>