Most teams that ship an LLM feature end up watching the wrong dashboard. They track uptime, P50 latency, and request volume — the same three metrics they'd track for a REST API — and call it observability. Then a model silently gets quieter about its own uncertainty, token costs creep up 3x in a quarter, or a tool-calling loop starts failing 1 in 5 times, and none of it shows up as a red number anywhere.
In 2025, Grafana Labs' Observability Survey found that just 7% of organizations run AI observability "in production, extensively, or exclusively" — 47% are still investigating the problem or stuck at proof-of-concept (Grafana Labs, 2025). Most teams haven't built the dashboard yet. This guide is a practical framework for the LLM production metrics that belong on it.
Key Takeaways
- In 2025, only 7% of organizations run AI observability in production; 47% are still investigating or piloting it (Grafana Labs, 2025)
- GPT-4's accuracy on a fixed task swung from 84% to 51.1% over a 3-month window with zero code changes on the caller's side (Stanford/UC Berkeley, 2023)
- In 2026, 73% of AI projects still blow their budget, and granular cost monitoring is the #1 most-requested FinOps capability for AI workloads (FinOps Foundation, 2026)
- Five metric categories matter for LLM systems: cost, quality drift, reliability, latency shape, and escalation rate — generic APM covers none of them well
If you haven't instrumented traces yet, start with OpenTelemetry for LLM applications — everything in this guide assumes you have gen_ai.* spans to query. This post covers what to do with that data once it exists.
Why Don't Standard APM Metrics Work for LLM Systems?
Standard APM tools were built to answer "is the service up and is it fast?" LLM systems fail in ways that question doesn't cover — a healthy-looking 200ms response can still be wrong, expensive, or degrading in accuracy relative to last month, and none of that shows up in an HTTP status code.
The gap is visible in what teams actually instrument. Grafana's 2025 survey found metrics adoption at 95% and logs at 87%, but distributed tracing — the telemetry type that actually captures an agent's step-by-step reasoning — sits at just 57% (Grafana Labs, 2025). Teams are collecting plenty of data. They're collecting the wrong shape of it.
That gap matters because uptime and latency are necessary, not sufficient. A model that answers instantly and confidently can still be wrong. The rest of this guide covers the five metric categories that catch what uptime dashboards miss: cost, quality drift, reliability, latency shape, and escalation rate.
What LLM Production Metrics Should You Actually Track?
No single dashboard needs all of this on day one. But five categories cover the failure modes that generic APM misses, and each one maps to a question your on-call engineer needs answered at 2 a.m.
| Category | Core Metric | The Question It Answers |
|---|---|---|
| Cost | Cost per trace, token usage trend | Is this feature about to blow the monthly budget? |
| Quality drift | Eval score over time, hallucination rate | Did the model get worse without anyone changing code? |
| Reliability | Error rate by type, tool failure rate | Is this a transient blip or a systemic failure? |
| Latency shape | P95/P99, time-to-first-token | Are users actually experiencing the P50 we report? |
| Escalation | Human override / fallback rate | Is the system quietly failing in a way users work around? |
An LLM gateway is the natural place to compute most of these — every request already flows through it, so cost, token counts, and error type are available at the point of aggregation instead of scattered across services. Each category below covers the specific metrics, why they matter, and what a reasonable alert threshold looks like.
Cost Metrics: Cost Per Trace and Token Growth
In 2026, granular AI cost monitoring — tracked by token, by LLM request, by GPU-hour — became the single most-requested capability among FinOps practitioners, according to the FinOps Foundation's State of FinOps 2026 report (n=1,192 practitioners tracking over $83B in cloud spend). The same report found 98% of organizations now actively manage AI spend, up from just 31% two years earlier (FinOps Foundation, 2026). Spend went from an afterthought to the top request in two budget cycles.
Tracking spend doesn't mean controlling it. As of 2026, 73% of AI projects still blow their budget, and wasted infrastructure spend rose to 29% of total AI cost, up from 27% the year before (FinOps Foundation, 2026). Total monthly spend is a lagging indicator — by the time it moves, the overrun already happened. Cost per trace, tracked continuously and grouped by feature and model, catches the overrun while it's still a trend line instead of an invoice.
The trend itself is accelerating. In 2026, Datadog's State of AI Engineering report found that median token usage per LLM request more than doubled year-over-year, and the heaviest-usage organizations — the 90th percentile — saw roughly a 4x increase (Datadog, 2026). If your dashboard only shows total dollars spent, you won't see which feature's context window is quietly growing until the bill reflects it.
If your cost dashboard stops at "total spend this month," you're reading the same lagging number the 73% figure describes. For the mechanics of cutting that number once you've found it, see prompt caching and model routing; for a full cost breakdown by category, the true cost of running AI agents at scale covers where the money actually goes beyond the API bill.
Quality and Drift Metrics: The Ones Everyone Skips
Here's the uncomfortable finding for anyone who assumes a model's accuracy is fixed once it ships: researchers at Stanford and UC Berkeley tracked GPT-4's accuracy on a simple prime-number identification task and found it dropped from 84% to 51.1% over a three-month window — with no code changes on the caller's side (Stanford/UC Berkeley, 2023). The model didn't get worse because anyone touched the prompt. It got worse because the vendor's underlying model changed underneath a stable API contract.
That's the core argument for tracking eval score as a metric, not a one-time gate. Most teams run evals before deployment and never again. Without a continuous eval score plotted against time, a silent regression like the one above looks like a normal Tuesday until support tickets pile up.
It gets stranger. A 2026 study found that even at temperature 0.0 — the setting most teams assume guarantees identical output for identical input — both GPT-4o-mini and Llama3.1-8b produced measurable non-deterministic drift across repeated identical prompts (arXiv, "Quantifying Non-Deterministic Drift in Large Language Models", 2026). Rerunning the same eval suite twice won't necessarily give you the same score. Track a rolling average, not a single run.
The assumption this breaks: most teams treat "temperature 0" as a determinism guarantee and stop worrying about output drift once it's set. The 2026 finding means that assumption is false at the hardware level — floating-point non-associativity in batched GPU inference produces different results for identical requests even with temperature locked to zero. Eval drift monitoring has to run continuously regardless of your temperature setting, not just after a prompt or model change.
Two more numbers worth tracking directly. Gartner's May 2026 prediction that 40% of organizations deploying AI will use AI observability to monitor model performance by 2028 signals where the industry is heading, not where it is today (Gartner, 2026). And on the hallucination side specifically, Vectara's Hallucination Leaderboard shows the best frontier models now hold grounded-summarization hallucination rates of 0.7%–1.5%, while most production models cluster between 1.8% and 6% (Vectara Hallucination Leaderboard, 2026) — a wide enough spread that "which model" is itself a metric decision.
A related trap: benchmark saturation. A 2026 systematic review of benchmarks released by OpenAI, Anthropic, Google, Meta, and Alibaba between January 2022 and November 2025 found widespread saturation — public benchmarks stop distinguishing model quality once every vendor optimizes against them (arXiv, "When AI Benchmarks Plateau", 2026). A public leaderboard score isn't a substitute for your own eval set against your own traffic. For the mechanics of building that eval loop, see how to evaluate your LLM agent.
Reliability Metrics: Error Rate by Type, Not Just Error Rate
A single "error rate" number flattens three very different failures into one line: rate limits, timeouts, and tool-call failures each need a different fix, and lumping them together means your on-call engineer opens the wrong runbook. If your gateway already exposes token counts and status codes, splitting error rate by error.type is a group-by, not a new instrumentation project — see LLM gateway architecture for where that field gets set.
Rate limits deserve their own alert, since they're usually the largest single error category and the most fixable — the fix is rate limiting and backoff strategy, not a model change. Tool-call failure rate, grouped by gen_ai.tool.name, surfaces the specific integration that's degrading, since a flaky search API and a flaky code-execution sandbox fail for unrelated reasons and need unrelated fixes.
Escalation rate — how often the system falls back to a human, a cached response, or a "let me connect you with support" message — is the metric most dashboards skip entirely, and it's the one that best reflects what users actually experience. A system with a low technical error rate but a rising escalation rate is still failing; it's just failing quietly, one deflected conversation at a time.
Latency Metrics Beyond P50
A P50 latency number hides the two populations that make it up. Prompt-cache hits return in well under a second; cold calls with a full context window can take several seconds. Averaging them produces a number that describes neither population accurately — which is exactly the blind spot standard APM dashboards create by default.
Two splits fix this. First, report P95 and P99, not just P50, and split each by cache-hit status — the two curves tell genuinely different stories, and conflating them hides the tail latency that actually drives complaints. Second, for any streaming interface, track time-to-first-token (TTFT) separately from total response time. A user judges responsiveness by how fast the first word appears, not by how fast the full answer finishes — two requests with identical total latency can feel completely different if one starts streaming in 200ms and the other in 3 seconds.
Token growth compounds this. As context windows fill up over the course of an agent loop, both latency and cost climb together — context engineering for AI agents covers how that bloat develops before it's visible on either dashboard. Tracking input-token count per iteration alongside latency turns a mystery slowdown into a chart you can point to in standup.
From Metrics to Alerts: Building the Dashboard
Collecting a metric and alerting on it are different projects, and most teams stop at the first one. A reasonable starting set of thresholds, assuming you've instrumented traces per the OpenTelemetry setup:
| Metric | Suggested Alert | Why This Threshold |
|---|---|---|
| Cost per trace | >2x rolling 7-day median | Catches a runaway prompt or model swap before the invoice does |
| Error rate (rate limits) | >5% of requests | Below this, retries usually absorb it silently |
| Tool failure rate | >5% for any single tool | Isolates a specific broken integration, not a general blip |
| Eval score (rolling) | >10% drop from 30-day baseline | Distinguishes real drift from single-run noise given non-determinism |
| Escalation rate | >2x rolling 7-day median | Rising fallback rate signals quiet degradation before errors spike |
| P99 latency | Above your streaming UX budget | P50 rarely reflects what the slowest 1% of users experience |
Group every one of these by model and by feature, not just globally. A global average can look flat while one specific feature quietly degrades — averaging across features is how a 51.1%-accuracy regression on one workflow disappears into a healthy-looking fleet-wide number.
In practice, the most common mistake in dashboards built this way isn't missing a metric — it's setting every alert threshold on day one, before there's a baseline. A cost-per-trace alert set against a guessed number either fires constantly and gets muted, or never fires and misses the real spike. Run each metric for at least a week to establish a rolling baseline before wiring the alert.
Frequently Asked Questions
What's the single most important metric for LLM systems in production?
There isn't one — that's the point of this guide. Cost per trace catches budget overruns, eval score catches silent quality drift, and error rate by type catches reliability issues, and each misses what the others catch. Teams that pick one "north star" metric for an LLM system consistently get blindsided by whichever category they didn't track.
How often should I run continuous evals against production traffic?
Daily at minimum, given that a 2026 study found measurable output drift even at temperature 0.0 (arXiv, 2026). Track a rolling average score rather than trusting any single run, and alert on a sustained drop from baseline rather than a single day's dip — see how to evaluate your LLM agent for building that pipeline.
Is cost per request enough, or do I need cost per trace?
Cost per trace, not per request. A single agent run can span a model call, a retrieval step, and several tool calls — cost per request only captures one leg of that chain. In 2026, granular cost monitoring by token and by request became FinOps teams' top-requested AI capability precisely because request-level tracking wasn't granular enough (FinOps Foundation, 2026).
Do these metrics apply the same way to single-call LLM features and multi-step agents?
Mostly yes, with more weight on reliability and escalation-rate metrics for agents. A single LLM call has one failure point; a multi-step agent has one per tool call, so tool failure rate and escalation rate matter proportionally more the longer the loop runs. Cost and quality-drift tracking apply identically to both.
How do I know if my hallucination rate is acceptable?
Compare it against your model tier, not an absolute number. Vectara's leaderboard shows 0.7%–1.5% for the best frontier models versus 1.8%–6% for most production models (Vectara Hallucination Leaderboard, 2026) — if you're well outside your tier's range, the fix is usually retrieval grounding or model choice, not prompt tweaking.
Conclusion
Uptime and P50 latency tell you the service is running. They don't tell you it's working — a model can be fast, available, and quietly wrong, over budget, or degrading in accuracy, and none of that trips a standard APM alert. The five LLM production metrics categories here — cost, quality drift, reliability, latency shape, and escalation rate — close that gap.
Start with whichever category is currently invisible to you. For most teams that's quality drift, since it's the one metric that requires deliberate, continuous measurement rather than a metric your existing stack already emits by default.
If you haven't wired up the trace data these metrics depend on, OpenTelemetry for LLM applications is the prerequisite setup guide — everything in this post assumes gen_ai.* spans already exist to query.
Sources
- Grafana Labs, Observability Survey 2025, retrieved 2026-07-12, https://grafana.com/observability-survey/2025/
- Stanford University / UC Berkeley, How Is ChatGPT's Behavior Changing over Time?, retrieved 2026-07-12, https://arxiv.org/abs/2307.09009
- Vectara, Hallucination Leaderboard (HHEM), retrieved 2026-07-12, https://github.com/vectara/hallucination-leaderboard
- Gartner, Gartner Predicts 40% of Organizations Deploying AI Will Use AI Observability to Monitor Model Performance by 2028, retrieved 2026-07-12, https://www.gartner.com/en/newsroom/press-releases/2026-05-12-gartner-predicts-40-percent-of-organizations-deploying-ai-will-use-ai-observability-to-monitor-model-performance-by-2028/
- FinOps Foundation, State of FinOps 2026, retrieved 2026-07-12, https://data.finops.org/
- arXiv, Quantifying Non-Deterministic Drift in Large Language Models, retrieved 2026-07-12, https://arxiv.org/abs/2601.19934
- arXiv, When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation, retrieved 2026-07-12, https://arxiv.org/abs/2602.16763
- Datadog, State of AI Engineering 2026, retrieved 2026-07-12, https://www.datadoghq.com/state-of-ai-engineering/
Related Posts
Weekly Digest
Get the best AI engineering posts, weekly
No hype. Curated signal every Sunday.