Agent Observability in 2026: Tracing, Evals, and Cost Attribution for Production Agents

The 2024–2025 cycle was about proving agents could work. Teams built prototypes, ran demos, and shipped experimental copilots. By 2026, the question has changed. Agents now run in production against real budgets, real SLAs, and real customers — and the teams operating them are discovering that shipping an agent is the easy part. Keeping it healthy, accurate, and affordable is the hard part.

The core thesis of this article is simple: agent observability is not one tool or one dashboard. It is three intertwined disciplines — LLM tracing, evals, and cost attribution — and teams that instrument only one of them fail at scale. You can have beautiful traces and no idea whether outcomes are correct. You can have rigorous evals and no visibility into which tenant is burning your token budget. You can have perfect cost dashboards and no way to explain why a run went sideways.

This guide breaks down why traditional APM falls short for AI agents, how to capture spans and trajectories, how to move evals into production, how to attribute LLM cost to the right owner, and how to sequence an agent observability rollout over 90 days.

Why Traditional APM Breaks Down for Agents

Standard application performance monitoring answers a narrow set of questions: how slow, how many errors, how much throughput. Those metrics still matter, but for agents they are necessary and nowhere near sufficient.

An agent makes multi-step decisions. It plans, calls tools, reads results, revises, and produces an outcome. A request can return HTTP 200 in 400ms and still be a catastrophic failure — the agent confidently answered the wrong question, hallucinated a tool argument, or looped through the same tool call six times before giving up. Traditional APM sees a healthy request. Your customer sees a broken product.

Agent systems introduce failure modes that infrastructure metrics simply cannot see:

• Tool-call loops where the agent retries the same failing action without escalating.
• Hallucinated tool arguments that pass schema validation but are semantically wrong.
• Context-window truncation that silently drops the instructions the agent needed.
• Silent retries that inflate latency and cost without surfacing an error.
• Prompt drift after a model upgrade changes behavior in ways no error code captures.

The lesson for 2026 is that agent observability must capture semantic correctness, not just infrastructure health. You need to know whether the agent did the right thing, not only whether the service stayed up.

"The most expensive agent failure is the one that returns 200 OK. Traditional APM will never page you for a wrong answer."

LLM Tracing: Spans, Trajectories, and Tool Calls

LLM tracing is the foundation everything else sits on. If you cannot reconstruct what an agent did, you cannot evaluate it or price it.

What to Capture Per Span

Every span in an agent run should carry enough context to be independently useful. At minimum, capture the prompt, completion, model and version, token counts (input and output), latency, tool name, tool arguments, tool result, and parent-child relationships. Model version matters more than teams expect — in our field work, silent provider model upgrades are a recurring cause of unexplained behavior shifts, which is why pinning and recording the exact model version per span is non-negotiable.

Trajectory-Level Tracing

A single span tells you one step. A trajectory tells you the story: the full agent run reconstructed as a graph, grouped by session and conversation. In the happy path a trajectory is a tree (or a DAG when steps fan out and rejoin), but real trajectories frequently contain cycles — retry loops, self-correction passes, tool re-invocations. Your trace model must represent those cycles explicitly rather than forcing them into a DAG, because the loops are precisely where cost and latency blow up.

Trajectory tracing is what lets you answer "where did it go wrong" rather than "what did this one call return," and it is the input that makes trajectory-level evals possible later.

OpenTelemetry Alignment

The OpenTelemetry GenAI semantic conventions are the right direction and are stabilizing, but treat them as still evolving, not frozen. Attribute names and span shapes have changed across releases. The pragmatic 2026 posture is:

• Instrument against OTel conventions, but pin the convention version your SDK emits and test upgrades deliberately.
• Keep a translation layer between your instrumentation and any proprietary backend. OTLP is a transport, not a guarantee — some vendors ingest OTel natively, others have their own native formats and partial OTel support. Vendor-neutral emission lets you switch backends without re-instrumenting.

[ILLUSTRATION: A single agent run rendered as a trajectory graph — tree with an explicit retry cycle — with spans for planning, tool calls, and final response, annotated with latency and token counts per node]

Pair standardization with a deliberate sampling strategy. Capture 100% of error runs and high-cost runs, and use tail-based sampling to control volume on the happy path. Tail-based sampling decides after the full trace completes (so you can keep traces that errored or exceeded a cost threshold), which requires a collector that buffers spans until the trace closes — plan your collector topology accordingly. Finally, design PII redaction and data retention up front. Traces contain prompts and tool results, which means they contain customer data — treat the trace store with the same compliance rigor as your primary datastore.

Evals in Production: From Offline Benchmarks to Online Signals

Tracing tells you what happened. Evals tell you whether it was good. The common 2025 pattern — a one-time benchmark run before launch — does not survive contact with production. In 2026, evals are a continuous production signal wired to alerting.

The Eval Pyramid

Structure evals in layers:

• Unit evals — does a single prompt or tool call produce the right output?
• Trajectory evals — did the multi-step path take a correct, efficient route?
• End-to-end task evals — did the agent actually accomplish the user's goal?

Each layer catches different failures. Unit evals catch prompt regressions fast and cheaply. Trajectory evals catch wasteful loops and wrong tool selection. End-to-end evals catch the outcome failures that matter most to customers.

Offline to Online

Start with golden datasets in CI so every prompt or model change runs against a fixed, versioned suite before merge. Then move the same graders into production as online evals that score a sample of live traffic continuously.

Three practices separate evals that work from evals that become noise:

1. Calibrate LLM-as-judge against human labels. Keep a small human-labeled set and measure judge agreement on it. A judge you have not calibrated is a random number generator with good manners. Record the judge model and version per score — a judge upgrade is a metric change and must be treated as such.
2. Sample deliberately. You do not need to score every request. Score 100% of failures and high-value or high-risk sessions, and a statistically adequate sample of the rest. This keeps eval cost bounded.
3. Wire scores to alerts with thresholds. An eval that writes to a dashboard is a report. An eval that fires when task-success rate drops below a threshold — or when judge disagreement spikes — is a production signal. Alert on rate of change, not just absolute value, so you catch regressions the day a model or prompt ships.

Cost Attribution: Who Burned the Tokens, and Why

Cost is the observability dimension teams most often skip, and the one that gets them in trouble with finance and customers first. Two questions get conflated:

• Allocation — which tenant, team, feature, or customer should this spend be charged to?
• Attribution — what actually drove the spend (which step, retry, or prompt)?

You need both. Allocation without attribution tells you a customer is expensive but not why. Attribution without allocation gives you interesting charts and a bill you cannot split.

Make Every Span Cost-Bearing

Compute cost at the span level, then roll up. Per span, record input tokens, output tokens, cached input tokens (billed at a different rate), and where applicable reasoning tokens (often billed as output). Multiply by the per-model price at the time of the call — store the price table version alongside the span so historical cost does not silently change when a provider updates pricing.

The Token Accounting Traps

• Retries double-count. A silent retry re-sends the prompt; if you sum span tokens you get the true provider bill, but if you dedupe you undercount. Decide explicitly and document it.
• Cached tokens are not free. They are cheaper, not zero. Price them separately.
• Provider rounding and minimums mean your computed cost will not exactly match the invoice. Reconcile against provider billing exports monthly and track the delta.
• Context growth is the quiet killer. Multi-turn agents re-send growing context each step; cost per turn is not constant. Chart tokens per step within a trajectory, not just per request.

The North-Star Metric: Cost per Successful Task

Raw token spend is a vanity metric. The number that matters is cost per successful task — total spend divided by tasks that actually succeeded, measured by your end-to-end evals. A change that doubles token usage but lifts success rate from 60% to 90% may be a bargain; a change that trims cost while quietly dropping success is a false economy. Tag every span with tenant_id, feature, session_id, and task_id so this metric can be sliced by owner.

Sequencing a 90-Day Rollout

Do not try to ship all three disciplines at once. Sequence them so each phase produces a signal the next phase depends on.

Days 1–30 — Instrument. Emit structured spans with the full per-span schema, pinned model versions, and tenant/feature tags. Stand up the trace store with retention and redaction policy. Success criterion: you can reconstruct any failed run end to end.

Days 31–60 — Evaluate. Build the golden dataset from real traces (especially failures), wire unit and trajectory evals into CI, and stand up online end-to-end evals on a sample of live traffic with calibrated judges. Success criterion: a prompt or model change that regresses task success is caught before merge and in production.

Days 61–90 — Attribute cost. Add span-level cost with a versioned price table, reconcile against the provider invoice, and publish cost per successful task sliced by tenant and feature. Success criterion: finance and product can both answer "what did this cost and was it worth it" without asking engineering.

The Bottom Line

Tracing, evals, and cost attribution are not three products to buy — they are three questions you must be able to answer about every agent run: What did it do? Was it right? What did it cost? Instrument all three, sequence the rollout, and treat each as a production signal rather than a report. Teams that do will ship agents they can actually defend to customers, finance, and their own on-call engineers.