In 2026, Datadog's State of AI Engineering report found that 5% of all LLM spans in production end in error — and 60% of those errors are rate limits (Datadog, 2026). Most teams find out about these failures from user complaints, not their observability stack.
The reason is structural. Standard APM tools — Datadog APM, New Relic, Dynatrace — were built for HTTP request/response cycles and database queries. They don't capture what matters for LLMs: token counts, model latency distributions, prompt caching hits, or the parent-child span hierarchy of a multi-step agent run. When a LangGraph agent fails on step three of seven, your current traces show a 500ms gap and nothing more.
OpenTelemetry fixes this. The project's official gen_ai.* semantic conventions — actively developed in the open-telemetry/semantic-conventions-genai repository and supported by Datadog, Honeycomb, and Grafana from version 1.37 onward — define a standard way to instrument any LLM provider. In May 2026, OpenTelemetry graduated to CNCF Graduated status, the highest maturity level, with 41% of organizations already running it in production (CNCF, 2026).
This guide walks through a complete OTel setup for production LLM applications: from a raw Anthropic SDK call to a full agent with tool calls and retrieval spans, exportable to Jaeger (local), Datadog, Honeycomb, or Grafana Tempo.
Key Takeaways
- In 2026, 5% of LLM production spans fail; 60% of those errors are rate limits that standard APM doesn't surface clearly (Datadog State of AI Engineering, 2026)
- OpenTelemetry's
gen_ai.*semantic conventions define 12+ standard attributes for LLM spans — input tokens, output tokens, model name, finish reason — available since OTel 1.27- A production-ready OTel setup requires four span types: model call, tool call, retrieval, and agent loop — each with different attribute sets
- Exporting to Jaeger takes 10 minutes locally; switching to Datadog or Honeycomb is a one-line exporter swap
If your token counts are already climbing and you haven't set up tracing yet, start with context engineering for AI agents — it covers how context bloat develops before it becomes visible in your costs.
Before You Begin
What you'll need:
- Python 3.9+
opentelemetry-api>=1.27andopentelemetry-sdk>=1.27anthropic>=0.26(Python SDK examples)- Docker (for running Jaeger locally — optional but recommended)
- Basic familiarity with distributed tracing concepts
Time: ~45–60 minutes
Difficulty: Intermediate
pip install opentelemetry-api opentelemetry-sdk \
opentelemetry-exporter-jaeger \
opentelemetry-exporter-otlp \
anthropicIf you're using LangChain or LangGraph, the Traceloop OpenLLMetry library auto-instruments both frameworks with a single import:
pip install traceloop-sdkStep 1: Why Standard APM Breaks for LLM Applications
Standard APM captures what happened at the network layer. LLM applications fail for reasons that live below that layer — and above it.
Token counts don't map to bytes. A 500ms response that consumed 8,000 input tokens costs 16× more than a 500ms response that consumed 500. Standard APM sees two spans with identical duration and HTTP 200 status. You see two "healthy" requests. Your finance team sees a bill that doesn't match your request volume.
Agent spans have nesting that HTTP spans don't. A LangGraph agent makes a model call → which emits two tool calls → which each trigger a retrieval span → which loop back to another model call. Standard APM captures those as five unrelated spans. OTel's trace context propagation keeps them in a single parent-child tree, so you can see total cost per agent run, failure point per step, and which tool call caused the retry cascade.
Latency distributions are bimodal. Prompt cache hits complete in under 100ms. Cold calls complete in 800–4,000ms. A single P99 hides both populations. You need to split by gen_ai.request.model and cache status — which is only possible if those attributes exist on the span.
In 2026, Datadog's State of AI Engineering report documented this blind spot directly: rate-limit errors account for 60% of all LLM errors, yet most engineering teams have no alerting on them because their APM tools classify 429s identically to any other client error, with no retry-cascade context (Datadog, 2026).
For teams already managing multi-provider traffic through an LLM gateway, OTel traces belong at the gateway layer — instrument it once and all downstream services inherit observability automatically.
Step 2: Install OpenTelemetry and Configure the gen_ai.* Conventions
OpenTelemetry's gen_ai.* semantic conventions define the attributes every LLM span should carry. They're provider-agnostic: whether you're calling Anthropic, OpenAI, or a self-hosted Llama model, the attribute names stay the same. The spec is currently experimental — actively evolving through versions 1.37–1.41 in the open-telemetry/semantic-conventions-genai repository — but major backends (Datadog, Honeycomb, Grafana Tempo) already support it natively from v1.37 onward, making it safe to adopt in production today.
The core attributes for model call spans:
| Attribute | Type | Description | Example |
|---|---|---|---|
gen_ai.system |
string | Provider identifier | "anthropic" |
gen_ai.operation.name |
string | Operation type | "chat" |
gen_ai.request.model |
string | Model requested | "claude-sonnet-4-6" |
gen_ai.response.model |
string | Model that responded | "claude-sonnet-4-6" |
gen_ai.usage.input_tokens |
int | Prompt token count | 1240 |
gen_ai.usage.output_tokens |
int | Completion token count | 340 |
gen_ai.request.max_tokens |
int | Max tokens setting | 4096 |
gen_ai.request.temperature |
float | Temperature | 0.7 |
gen_ai.response.finish_reasons |
string[] | Why generation stopped | ["end_turn"] |
error.type |
string | Error class on failure | "rate_limit_error" |
Set up your TracerProvider once at application startup:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
def setup_tracing(service_name: str = "my-llm-app"):
jaeger_exporter = JaegerExporter(
agent_host_name="localhost",
agent_port=6831,
)
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))
trace.set_tracer_provider(provider)
return trace.get_tracer(service_name)
tracer = setup_tracing()For production, swap JaegerExporter for OTLPSpanExporter — a one-line change shown in Step 5.
According to the OpenTelemetry specification, gen_ai.* conventions are designed to work alongside existing HTTP and RPC conventions, not replace them (OpenTelemetry GenAI Semantic Conventions, 2026). Your existing HTTP trace for the /api/chat endpoint can parent the gen_ai span — giving you end-to-end visibility from user request to model response in a single trace tree.
One easily-missed finding from the Datadog 2026 report: only 28% of LLM call spans used prompt caching despite broad provider support (Datadog, 2026). With token-aware tracing in place, you can finally see which calls are caching and which aren't — and quantify the cost delta. Teams that enabled caching and deduplicated retrieval queries reduced token spend by over 30% and cut p95 latency in half, according to the same report.
Step 3: Instrument Your First LLM Call
By the end of this step, you'll have a working trace for a single Anthropic SDK call — with token counts, model name, and finish reason captured as span attributes.
import anthropic
from opentelemetry import trace
client = anthropic.Anthropic()
tracer = trace.get_tracer("my-llm-app")
def call_claude(prompt: str, model: str = "claude-sonnet-4-6") -> str:
with tracer.start_as_current_span("gen_ai.chat") as span:
# Set request attributes before the call
span.set_attribute("gen_ai.system", "anthropic")
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.request.max_tokens", 1024)
try:
response = client.messages.create(
model=model,
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
)
# Set response attributes after the call
span.set_attribute("gen_ai.response.model", response.model)
span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
span.set_attribute("gen_ai.response.finish_reasons", [response.stop_reason])
return response.content[0].text
except anthropic.RateLimitError as e:
span.set_attribute("error.type", "rate_limit_error")
span.set_status(trace.StatusCode.ERROR, str(e))
raise
except anthropic.APIError as e:
span.set_attribute("error.type", type(e).__name__)
span.set_status(trace.StatusCode.ERROR, str(e))
raiseVerify it works: Start Jaeger locally and call your function:
docker run -d -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one
python -c "from your_module import call_claude; call_claude('Hello')"Open localhost:16686, find the my-llm-app service, and confirm the span shows gen_ai.usage.input_tokens and gen_ai.usage.output_tokens in the tags panel.
When your trace carries token counts, cost attribution becomes a query: (input_tokens / 1_000_000) * input_price + (output_tokens / 1_000_000) * output_price. Store this as a derived metric in your backend and you've replaced the monthly invoice surprise with a per-request cost dashboard.
Step 4: Add Multi-Step Agent Spans (Tool Calls and Retrieval)
Single model call spans are the easy part. The real value of OTel for LLM applications is tracing the full agent graph — where a failure in tool call three cascades into four retries, each burning tokens, before surfacing as a user-visible error.
For a well-instrumented agent, you need four span types in a parent-child hierarchy:
[agent.run span] ← parent: full run cost + total duration
├── [gen_ai.chat span] ← model call: tokens, model, finish reason
│ └── [gen_ai.tool_call span] ← tool call: tool name, input params, result
└── [gen_ai.retrieval span] ← retrieval: query, doc count, latencyHere's how to instrument this hierarchy manually (works with any framework):
def run_agent(user_query: str) -> str:
with tracer.start_as_current_span("agent.run") as agent_span:
agent_span.set_attribute("agent.query", user_query[:200])
total_input_tokens = 0
total_output_tokens = 0
for iteration in range(10): # max iterations guard
agent_span.set_attribute("agent.iteration", iteration)
# Model call span — child of agent span
with tracer.start_as_current_span("gen_ai.chat") as model_span:
model_span.set_attribute("gen_ai.system", "anthropic")
model_span.set_attribute("gen_ai.request.model", "claude-sonnet-4-6")
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=build_messages(user_query, iteration),
)
model_span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
model_span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
total_input_tokens += response.usage.input_tokens
total_output_tokens += response.usage.output_tokens
# Instrument each tool call as a child span
for tool_call in extract_tool_calls(response):
with tracer.start_as_current_span("gen_ai.tool_call") as tool_span:
tool_span.set_attribute("gen_ai.tool.name", tool_call["name"])
tool_span.set_attribute("gen_ai.tool.call.id", tool_call["id"])
result = execute_tool(tool_call)
tool_span.set_attribute("gen_ai.tool.result_length", len(str(result)))
if response.stop_reason == "end_turn":
break
# Capture cumulative cost on the parent span
agent_span.set_attribute("agent.total_input_tokens", total_input_tokens)
agent_span.set_attribute("agent.total_output_tokens", total_output_tokens)
return response.content[0].textThe non-obvious detail: Always capture agent.total_input_tokens on the parent agent.run span, not only on individual model call children. When debugging a cost spike, you want to filter by agent-level cost — not sum across dozens of child spans in your query language. This one attribute turns cost queries from O(n) span aggregations to O(1) attribute lookups.
If you're on LangChain or LangGraph, OpenLLMetry instruments the full span hierarchy automatically with a single call:
from traceloop.sdk import Traceloop
Traceloop.init(app_name="my-agent")
# All LangChain/LangGraph calls now auto-emit gen_ai.* spansAs of Q2 2026, OpenLLMetry supports LangChain, LangGraph, LlamaIndex, and raw OpenAI/Anthropic SDKs (Traceloop OpenLLMetry, 2026).
Step 5: Export Traces to Your Backend
Jaeger runs locally in a single Docker container — ideal for development. Production requires a more durable, scalable backend.
Local development with Jaeger:
docker run -d \
-p 6831:6831/udp \
-p 16686:16686 \
--name jaeger \
jaegertracing/all-in-one:latestNavigate to localhost:16686 to view traces.
Production — swap to the OTLP exporter. It's a one-line change:
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Replace JaegerExporter with:
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")Place an OTel Collector between your app and the backend. It decouples your application from vendor choices and lets you fan out to multiple backends:
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
timeout: 5s
# Tail-based sampling: always keep errors, keep 10% of successful spans
tail_sampling:
decision_wait: 10s
policies:
- name: errors-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: probabilistic-policy
type: probabilistic
probabilistic: {sampling_percentage: 10}
exporters:
datadog:
api:
key: ${DD_API_KEY}
honeycomb:
api_key: ${HONEYCOMB_API_KEY}
dataset: llm-production
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, tail_sampling]
exporters: [datadog, honeycomb]
Don't use uniform sampling on LLM traces. A flat 10% sample rate statistically drops 10% of your error traces — exactly the ones you need. The tail-sampling config above samples 100% of error spans and 10% of successful ones, keeping your storage costs predictable without sacrificing error coverage.
Per the OpenTelemetry SDK benchmarks, span creation and attribute setting adds roughly 0.5–2ms per span (OpenTelemetry Performance Benchmarks, 2025). For LLM workloads where model latency dominates at 500–4,000ms, this overhead is under 0.4% — effectively invisible to end users.
Step 6: Derive Actionable Metrics from Traces
Once your traces carry gen_ai.* attributes, four metrics become immediately queryable from your trace backend. These are the signals that separate "we have tracing" from "we have observability."
The context bloat trend — tracking gen_ai.usage.input_tokens across agent iterations — deserves special attention. In 2025, Chroma's context rot research found that all 18 tested frontier LLMs showed measurable performance degradation as input length grew, across every length increment tested (Chroma Context Rot Research, 2025). Tracing lets you catch this before it becomes a quality problem, not just a cost problem.
For teams already managing rate limit handling patterns, these traces add a missing dimension: when a rate limit fires, you can now see whether it correlated with a spike in input tokens (context bloat) or a burst in request rate — the root cause determines the fix.
Common Mistakes to Avoid
Most teams get tracing working and then undermine it with four predictable errors.
1. Skipping token attributes on error spans. When a rate-limit error fires, the most useful debugging question is: "How many tokens was that request?" Most instrumentation code skips gen_ai.usage.* on the exception path because it runs before the response arrives. Set whatever token data you have in a finally block — at minimum, log the requested model and max_tokens. Without this, you can't correlate token volume with rate-limit frequency, which is the data that tells you whether to cache or route to a different model.
2. Breaking trace context across async steps. In async Python, OpenTelemetry context isn't automatically propagated across asyncio.create_task() or concurrent.futures. You need to explicitly propagate it:
import contextvars
ctx = contextvars.copy_context()
await loop.run_in_executor(None, ctx.run, your_function)This is the most common reason agent spans appear as orphaned traces — spans exist, but they're disconnected in the UI. The fix is one line. Finding the bug costs hours.
3. Sampling everything uniformly. A flat 10% sample rate on LLM traces drops 10% of errors — exactly the traces you need. Use tail-based sampling rules (shown in Step 5): always keep error spans, always keep agent runs above a token threshold, sample routine successful calls at 10–20%.
4. Conflating LangSmith with OpenTelemetry. LangSmith is excellent for LangChain-only stacks, but it doesn't work outside that ecosystem and doesn't integrate with standard APM backends. OpenTelemetry is vendor-neutral — it works across any LLM provider and integrates with Datadog, Honeycomb, Grafana, or Jaeger. For multi-framework stacks, OTel is the right choice; LangSmith and OTel can run in parallel during migration.
What Success Looks Like
If everything went correctly, you now have:
- A trace UI showing agent runs as connected parent-child span trees (not orphaned fragments)
- Token counts on every model call span, queryable for cost-per-trace metrics
- Tool errors surfacing as child spans with
error.typeattributes - A sampling policy that keeps 100% of error traces and 10% of healthy ones
Your next steps: configure an alert on error.type = "rate_limit_error" — which now appears in your trace backend with full context — and build a cost-per-trace dashboard widget using the agent.total_input_tokens attribute from your agent-loop spans.
For teams building continuous evaluation pipelines, traces feed directly into the eval layer. The span data answers "did this agent call succeed?" while your evaluator answers "did it succeed correctly?" Read how to evaluate your LLM agent for how to close that loop.
Frequently Asked Questions
Does OpenTelemetry natively support LLM tracing?
Yes. OpenTelemetry added official gen_ai.* semantic conventions in version 1.27, covering attributes for model calls, token usage, finish reasons, and tool calls across all major LLM providers. The spec is maintained by the OpenTelemetry GenAI working group and is provider-agnostic — the same attribute names work for Anthropic, OpenAI, Google, and self-hosted models (OpenTelemetry Semantic Conventions — GenAI, 2026).
Can I use OpenTelemetry with the Anthropic Python SDK?
Yes. The Anthropic SDK doesn't auto-instrument for OTel, so you add spans manually as shown in Step 3, or use Traceloop's OpenLLMetry (pip install traceloop-sdk) which wraps the SDK and emits gen_ai.* spans automatically. Manual instrumentation gives you full control over attribute naming and span boundaries; auto-instrumentation ships faster with less code.
What's the difference between OpenTelemetry and LangSmith for LLM tracing?
LangSmith is LangChain's proprietary tracing platform — excellent for LangChain-only stacks, but it doesn't work outside that ecosystem and doesn't integrate with standard APM backends. OpenTelemetry is a vendor-neutral standard that works across any LLM provider and any backend (Datadog, Honeycomb, Grafana Tempo, Jaeger). For multi-framework or multi-provider stacks, OTel is the right choice. They can run simultaneously during migration.
How much latency does OpenTelemetry add to LLM applications?
Per the OpenTelemetry SDK performance benchmarks, span creation and attribute setting adds roughly 0.5–2ms per span (OpenTelemetry Performance Benchmarks, 2025). For LLM workloads where model calls take 500–4,000ms, this overhead is under 0.4%. The BatchSpanProcessor exports spans asynchronously, so trace export doesn't block your main request path.
Can I trace RAG retrieval calls with gen_ai.* conventions?
Yes. For a vector database call, create a gen_ai.retrieval child span under the model call that triggered it. Set gen_ai.retrieval.documents (count of returned documents) and include the query text as a span attribute. This gives you per-query retrieval latency broken down by which model call initiated the lookup — useful for identifying slow retrievals that inflate agent latency. If context bloat appears in your token trends, context engineering for AI agents covers the structural fixes.
Conclusion
You now have a production-ready OTel setup for LLM applications: token-aware spans for single model calls, a four-span-type hierarchy for agent graphs, and a one-line backend swap from Jaeger to Datadog or Honeycomb.
The 5% error rate and 60% rate-limit share documented for 2026 aren't going away — but they stop being invisible the moment your traces carry error.type attributes and your agent loops produce connected parent-child span trees.
If you're building out the full observability layer, the next two posts in this series cover continuous eval pipelines (closing the loop from traces to quality signals) and LLM metrics dashboards (translating span data into alertable SLOs).
Sources
- Datadog, State of AI Engineering 2026, retrieved 2026-07-05, https://www.datadoghq.com/state-of-ai-engineering/
- Datadog, State of AI Engineering 2026 Press Release, retrieved 2026-07-05, https://www.datadoghq.com/about/latest-news/press-releases/datadog-state-of-ai-engineering-report-2026/
- CNCF, OpenTelemetry Graduation Announcement, retrieved 2026-07-05, https://www.cncf.io/announcements/2026/05/21/cloud-native-computing-foundation-announces-opentelemetrys-graduation-solidifying-status-as-the-de-facto-observability-standard/
- Grafana Labs, 4th Annual Observability Survey 2025, retrieved 2026-07-05, https://grafana.com/observability-survey/2025/
- OpenTelemetry, GenAI Semantic Conventions, retrieved 2026-07-05, https://opentelemetry.io/docs/specs/semconv/gen-ai/
- OpenTelemetry, Gen AI Attribute Registry, retrieved 2026-07-05, https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/
- OpenTelemetry, SDK Performance Benchmarks, retrieved 2026-07-05, https://opentelemetry.io/docs/specs/otel/performance/
- OpenTelemetry, open-telemetry/semantic-conventions-genai GitHub Repository, retrieved 2026-07-05, https://github.com/open-telemetry/semantic-conventions-genai
- Traceloop, OpenLLMetry GitHub Repository, retrieved 2026-07-05, https://github.com/traceloop/openllmetry
- Chroma, Context Rot Research, retrieved 2026-07-05, https://trychroma.com/research/context-rot
- Datadog, LLM Observability Natively Supports OTel GenAI Semantic Conventions, retrieved 2026-07-05, https://www.datadoghq.com/blog/llm-otel-semantic-convention/
Related Posts
Weekly Digest
Get the best AI engineering posts, weekly
No hype. Curated signal every Sunday.