R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
Add the felhom-controller container as a special entry in the
app_telemetry array sent to the hub. This reuses all existing hub
infrastructure (storage, aggregation, UI) with zero hub-side changes.
The controller's memory/CPU metrics and log warnings/errors are now
collected alongside app telemetry, giving the hub visibility into
controller health, memory trends, and known issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deployed-but-stopped apps were included in telemetry reports with all-zero
memory/CPU values, dragging down hub-side averages. Now isStackRunning()
filters to only running/starting/unhealthy/restarting states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New GET /api/debug/telemetry endpoint runs full telemetry pipeline on-demand
- GetTelemetryPreview callback added to DebugCallbacks, wired in main.go
- BuildAppTelemetryForDebug() exported wrapper in report/telemetry.go
- Debug page: new collapsible section with per-app table (memory, CPU, log errors/warnings, issues) and raw JSON viewer
- Available regardless of hub configuration
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New internal/metrics/telemetry.go: MetricsStore.GetContainerTelemetry()
aggregates container memory/CPU from SQLite over the last 15 min
- New internal/metrics/logscanner.go: ScanContainerLogs() scans docker logs
for errors/warnings, deduplicates via fingerprinting (strips timestamps,
replaces 6+ digit numbers, hex strings, UUIDs)
- New internal/report/telemetry.go: buildAppTelemetrySection() assembles
per-stack AppTelemetry by aggregating container metrics and log summaries
- internal/report/types.go: added AppTelemetry field to Report struct plus
AppTelemetry type with memory/CPU/log fields and LogIssue references
- internal/report/builder.go: calls buildAppTelemetrySection() in BuildReport()
- Backward-compatible: old Hub versions silently ignore app_telemetry field
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
admin