Full-Stack Observability
for AI Agents

Why end-to-end in OTel only
hides too much

Matthew Khouzam | Ericsson
AAIF Observability Working Group | August 2026

Matthew Khouzam | Ericsson - AAIF Observability WG

The Thesis

  1. Know everything that exists - the full stack from silicon to agent
  2. Then select the subset appropriate for your use case
  3. Never start by limiting what you can see
You can't optimize what you can't observe.
You can't observe what you don't know exists.

Observability Is Foundational

Security, performance, and accuracy of a workflow all rely on observability.

  • Security - you can't detect anomalies, unauthorized access, or data exfiltration without visibility into what's happening
  • Performance - you can't optimize latency, throughput, or resource usage without measuring them across every layer
  • Accuracy - you can't validate that an AI workflow produces correct results without observing its intermediate steps and decisions
Without observability:
Security is blind guessing.
Performance is wishful thinking.
Accuracy is unverifiable.
Observability isn't a feature -
it's the foundation the other pillars stand on.

The AI Observability Stack

05 - AI Agent Observability
OTel, Langfuse, Datadog, TRACE attestation
04 - Network
Wireshark, MQTT, NCCL traffic
03 - Hardware Accelerators
NVIDIA Nsight, AMD rocprof, PyTorch Profiler
02 - Kernel Tracing
LTTng, perf, ftrace, eBPF, driver tracing
01 - Foundations
CTF binary format, Trace Compass analysis

Each layer builds on the one below. Skip a layer -> blind spot.

What OTel Sees

[yes] Strengths

  • Distributed traces across services
  • W3C Trace Context propagation
  • gen_ai.* semantic conventions
  • Vendor-neutral export pipeline
  • Metrics, logs, traces unified

[no] Gaps

  • No kernel-level context
  • No GPU hardware counters
  • No network packet attribution
  • No CFS throttling visibility
  • No hardware attestation
  • gen_ai.* still experimental
  • No MCP-aware propagation

The Hidden 95ms

A 100ms OTel span might be:
5ms computation + 95ms CFS throttle

Application-level spans report wall-clock time that includes all scheduling distortion:

  • CFS throttling - 100ms gaps per quota period, invisible to the app
  • vCPU pre-emption - hypervisor steals time, guest never knows
  • Noisy neighbors - runqueue contention inflates latency
  • GPU queueing - kernels waiting for SM allocation

Only kernel tracepoints (sched_switch, sched_stat_wait) can decompose wall-clock into CPU-time vs throttle vs wait.

What Each Layer Reveals

LayerRevealsOTel Equivalent
Agent (05)LLM calls, token usage, tool invocations[yes] gen_ai.* spans
Network (04)Wire-level: NCCL collectives, API latency, packet loss[no] None
GPU (03)Tensor core utilization, memory bandwidth, kernel stalls[no] None
Kernel (02)Scheduler decisions, CFS throttle, memory pressure, I/O[no] None
Foundations (01)Nanosecond precision, zero-copy binary traces, cross-layer sync[no] None

4 out of 5 layers are invisible to OTel-only observability.

Cross-Layer Synergies

When layers share context, you get answers none can provide alone:

  • Kernel sched_switch + pcap -> every packet attributed to a process (free)
  • Kernel sched_switch + Intel PT -> full per-process instruction traces
  • GPU tracepoints + kernel scheduler -> GPU commands attributed to owner processes
  • OTel spans + LTTng sched -> decompose span duration into CPU vs wait vs throttle
  • PyTorch Profiler + DCGM + kernel -> operator -> GPU kernel -> scheduling end-to-end
The kernel trace is the shared index. It knows which TID runs on which CPU at every context switch. Every other layer references this without duplicating context.

Overhead Reality

Full-stack doesn't mean full-overhead. Production-safe layered approach:

ToolOverheadMode
OTel SDK (agent spans)~0 ns disabled, 1-3% activeAlways-on
DCGM (GPU fleet monitoring)< 0.1%Always-on (1 Hz)
LTTng kernel (sched + I/O)1-3%Always-on flight recorder
AMD IBS / LBR (CPU profiling)< 1%Always-on sampling
NVTX/ROCTX annotations< 0.01%Compiled in, free without collector
perf stat (hardware counters)~0%Counting mode

Combined always-on overhead: < 5%. You can observe everything in production.

The OTel-Only Trap

Scenario

AI agent inference latency spikes from 200ms to 2s intermittently in production.

OTel shows: slow LLM span. That's it.

Full-stack shows:

  • Container was CFS-throttled (cpu.stat)
  • GPU memory was swapping (DCGM)
  • NCCL collective was stalled by a straggler rank
  • The straggler was pre-empted by a noisy neighbor VM

Root cause found in minutes, not days.

The Approach

Know everything. Then select.

  1. Instrument all layers - compile in tracepoints, annotations, probes
    Cost when disabled: literally zero (NOP patching, static branches)
  2. Always-on lightweight collection - flight recorders, 1Hz monitors, counting mode
    Cost: < 5% combined. You get continuous baseline.
  3. On-demand deep-dive - enable detailed collection when anomaly detected
    Nsight Compute, Intel PT, full PyTorch profiling - staging/dev only for these.
  4. Correlate across layers - Trace Compass, TMLL, or custom analysis
    The unified view is where root causes emerge.

What This WG Should Consider

  • Correlation IDs across layers - how does an OTel trace_id link to a kernel trace session?
  • Minimum observable surface - what MUST be observable for an AI agent deployment to be considered "instrumented"?
  • Privacy at emission - PII sanitization at the source, not the storage
  • Governance evidence - TRACE spec: hardware-attested records of what executed, where, under which policy
  • Subsetting guidance - which layers to activate for which use case (cost-optimized, latency-critical, compliance)

Key Takeaways

1. OTel is necessary but not sufficient - it's layer 5 of 5
2. 80% of root causes live below the application layer
3. Full-stack always-on observability costs < 5% total overhead
4. Know everything exists -> select the right subset -> never start blind

Resources

AAIF Reference Architecture Assessments

Full collection5 layers, 20+ technologies assessed
Best practices guidePractitioner's guide from silicon to agent
Evaluation dimensionsObservability, Security, Identity, Reliability, Accuracy

Matthew Khouzam | Ericsson
AAIF Observability Working Group | August 2026