AI-Powered Trace Analysis

Tracing & Trace Compass

Context before the code

What is Tracing?

• Recording timestamped events from a running system, kernel, userspace, network
• Non-intrusive: nanosecond overhead, always-on capable
• Produces massive datasets: a few minutes → gigabytes of events

What is Trace Compass?

• Open-source Eclipse project for trace visualization & analysis
• Supports LTTng, CTF, perf, ftrace, and more
• Powerful, but requires expertise to use effectively
• Trace Server Protocol (TSP) exposes analysis via REST API

Why Add MCP?

• Users shouldn't need to be tracing experts to get insights
• "Find CPU anomalies in this trace" is easier than navigating 20 views
• AI handles the how, the user focuses on the what

What's New in Trace Compass

• CTF2 support, JSON-based trace metadata, human-readable, AI-parseable
• Opens the door to smarter AI-driven analysis
• AI can read trace structure without binary parsing

TMLL, The ML Layer

• Anomaly detection, iforest, z-score, IQR, moving average, combined
• Change point analysis, single, z-score, voting, PCA
• Correlation analysis, Pearson, Spearman, Kendall
• Memory leak detection, idle resources, capacity planning

Python ML library on top of the Trace Server Protocol, powerful, but requires code to use

What is MCP?

Model Context Protocol, the universal adapter between AI and tools

MCP in 30 Seconds

• Open protocol: JSON-RPC over stdio or HTTP
• Write a tool once → use from any MCP-compatible agent
• Adopted by OpenAI, Anthropic, Google, all major IDEs

Why MCP Matters for You

Deterministic tools + AI reasoning = best of both worlds

Deterministic Execution

• AI decides what to call, your tool decides how to execute
• No hallucinated analysis, real code runs on real data
• Reproducible results: same input → same output, every time

Save Tokens

• Don't paste raw data into the prompt, let the tool process it server-side
• Return summaries, not megabytes of CSV
• Progressive discovery: only load schemas for tools you actually use (~80% savings)

Improve Your Existing Tools

• Your CLI already works, MCP makes it AI-accessible
• No rewrite needed: wrap, don't replace
• Users who can't write Python can now use your tool via natural language

Performance & Cost

Why not just feed the trace to the LLM?

Raw Trace → LLM vs MCP + TMLL

A 1 GB kernel trace ≈ billions of text tokens, exceeds every context window

• LLMs can't reliably do statistics on raw events
• MCP approach: AI sends one tool call, gets deterministic results back
• Trace Server + TMLL do the heavy lifting, AI just interprets

MCP Overhead

• ~1 s overhead per call (subprocess + Python startup)
• Negligible next to LLM inference latency
• Future: in-process mode eliminates the cost

Runs Locally, Privacy & Free Compute

• Trace Compass, TMLL, and the MCP server all run on your machine
• Your traces never leave the network, critical for regulated, proprietary, or customer data
• Uses compute you already own, no new cloud bill, no GPU rental
• The AI only sees the small, aggregated result, not the raw events

How to Start

Step 1: Create a CLI

Create a CLI

#!/usr/bin/env python3
"""tmll_cli.py, 12 subcommands wrapping the TMLL library."""
import argparse
from tmll.tmll_client import TMLLClient

def detect_anomalies(args):
    client = TMLLClient(args.host, args.port)
    experiment = get_experiment(client, args.experiment)
    outputs = experiment.find_outputs(keyword=args.keywords, type=['xy'])
    ad = AnomalyDetection(client, experiment, outputs)
    result = ad.find_anomalies(method=args.method)
    print(f"Found {total} anomalies across {len(result.anomalies)} outputs")

# ... 11 more subcommands ...

Standard argparse, nothing MCP-specific yet

Step 2: Wrap CLI in MCP

~615 lines of Python total

The MCP Wrapper

from mcp.server.fastmcp import FastMCP
mcp = FastMCP("tmll-cli-mcp-server")

@mcp.tool()
def detect_anomalies(experiment_id: str,
                     keywords: list[str] = None,
                     method: str = None) -> str:
    """Detect anomalies in trace data using ML methods."""
    args = build_args({
        "keywords": ("-k", keywords or ["cpu usage"]),
        "method": ("-m", method or "iforest"),
    })
    return run_cli("anomaly", experiment_id, *args)

Any CLI → MCP tool in 5 lines of glue

The Stack

Why MCP → CLI → Library?

• Separation of concerns, you can access and test your CLI independently, no AI needed
• MCP is new (2024), put it on tested footing by building on a proven CLI layer
• One path to maintain, why have two code paths when one works for both humans and AI?

12 Tools Exposed

MCP Apps

Rich content returned inline

Images in AI Output

@mcp.tool()
def plot_xy_with_anomalies(experiment_id: str,
                           as_image: bool = True):
    """Detect anomalies and return annotated charts."""
    # ... run analysis, generate matplotlib plot ...
    return Image(data=buf.getvalue(), format="png")

Debugging MCP

Theia IDE & MCP Inspector

See the Results

One MCP server, every AI client

Eclipse Theia & Gemini CLI

Theia, inline anomaly chart

Gemini, 1 GB trace, 85K tokens, 5 tool calls