LLM Observability at Scale: Tracing, Evals, and Cost in One Stack

Why Your LLM Dashboard Lies to You

A latency spike hits production on a Tuesday. Your APM dashboard shows p95 response time doubling, but the stack trace is clean — no exceptions, no timeouts, no failing dependencies. Six hours later you find it: someone edited a prompt template to add three few-shot examples, and the token count tripled. Nobody logged the change. Nobody connected it to the graph.

Or the quieter failure: your monthly provider invoice arrives well over budget. Finance asks which feature, which customer, which team drove it. You have no answer, because cost was never attached to anything but a billing account.

These are not monitoring failures. They are architecture failures. Tracing, evals, and cost are not three tools bolted together — they are one data problem wearing three hats. This article lays out a defensible LLM observability stack that treats a single trace ID as the join key across all three signals, and gives you a reference architecture, a build-vs-buy framework, and a 30/60/90 roadmap to get there.

Why Fragmented LLM Observability Breaks at Scale

Most teams ship LLM features with the tooling they already had: an APM vendor for latency, a notebook for evals, and a finance spreadsheet for spend. That works at 10,000 requests a day. It collapses at 10 million.

Three failure patterns show up, usually in this order.

Metrics without context. Your dashboard reports p95 latency by endpoint. It cannot tell you which prompt version, which retrieved document, or which tool call produced the slow tail. Aggregated metrics are excellent at telling you something is wrong and useless at telling you what.

Evals trapped in notebooks. A data scientist runs an offline eval suite against a golden dataset, gets a score, and reports it in a slide. Meanwhile production traffic drifts — new user segments, new query distributions, new failure modes — and the eval suite never sees any of it. Offline evals measure the world as it was when the dataset was frozen; without a continuous refresh from live traffic, that world diverges from production within weeks. (See: how to build an eval suite that runs in CI.)

Cost visible only at the invoice. The provider bill is one number. It has no dimension for feature, team, or tenant. When a single enterprise customer's usage doubles because they discovered your agent, you find out from finance, not from your platform.

The compounding factor is non-determinism. With a traditional service, a stack trace plus the input is usually enough to reproduce a bug. With an LLM pipeline, the same input produces different outputs across runs, model versions, temperature settings, and retrieval results. You cannot reproduce a bug from a stack trace alone. You need the entire execution context — every prompt, every retrieved chunk, every tool response, every model parameter — captured at the moment it happened.

That requirement is what forces consolidation. If traces live in one system, eval scores in another, and cost in a third, you will spend your engineering budget building joins instead of building product.

The single architectural claim of this article: every eval score and every cost line item should attach to a trace ID. If it doesn't, you don't have observability — you have three unrelated datasets.

The Three Signals of an LLM Stack

Strip away the vendor marketing and an LLM stack produces exactly three signals.

Tracing answers what happened. The full execution path of a request: what was retrieved, what tools were called, what the model saw, what it returned, how long each step took.
Evals answer was it good. Scored quality — correctness, groundedness, safety, tone, task completion — measured either against curated datasets or against live production traffic. (Related: offline vs. online evals: when to use each.)
Cost answers what it cost. Tokens in, tokens out, cached tokens, reasoning tokens, per-model rates, and the unit economics of a task or a customer.

They share one substrate: the trace. A trace is the only object that exists at the moment of execution and can carry all three signals forward. An eval is a function of a trace's inputs and outputs. A cost line item is a function of a trace's token counts and model routing. When you attach both to the trace ID, you can ask questions that no single tool can answer:

Which prompt version has the worst groundedness and the highest cost per resolved ticket?
Which tenant's traffic is driving the latency tail?
Did last week's model downgrade save money without hurting eval scores?

Those are the questions that justify the platform.

Tracing — Reconstructing a Single Request End-to-End

Spans, Trace Trees, and the GenAI Semantic Conventions

A trace is a tree of spans. Each span is a timed operation with a name, a start, an end, and a set of attributes. A typical agentic request looks like this:

request (root span)
├── retrieval.search
│   ├── vector.query
│   └── rerank
├── llm.call (planning)
├── tool.call: get_order_status
├── llm.call (synthesis)
└── postprocess.guardrail

Parent-child relationships give you the critical path. Attributes give you the payload: gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, prompt and completion content, tool arguments, and retrieval document IDs.

The portability layer here is the OpenTelemetry GenAI semantic conventions. They standardize attribute names for model, token usage, and operation type across providers. Instrumenting against the convention rather than a vendor SDK is your hedge against tool churn — if you switch backends in eighteen months, your instrumentation survives. (See: OpenTelemetry GenAI conventions: a practical guide.)

Two caveats an expert audience will hold you to. First, these conventions are still stabilizing — attribute names have been renamed across releases (for example, the shift from gen_ai.system to gen_ai.provider.name, and from gen_ai.usage.prompt_tokens to gen_ai.usage.input_tokens). Pin the semantic-conventions version you build against, and wrap emission in a thin internal adapter so a rename is a one-file change, not a codebase-wide migration. Second, the conventions cover the common fields; your domain-specific attributes (tenant ID, prompt version, feature flag, retrieval corpus version) are yours to define. Treat the OTel spec as the floor, not the ceiling.

This is not a theoretical benefit. Observability vendors consolidate, get acquired, and change pricing. Teams that instrumented against a proprietary SDK in 2024 spent 2025 rewriting their instrumentation. Teams that used the open convention swapped a config value.

What Every Span Must Carry

A trace is only as useful as its attributes. At minimum, every LLM span should carry:

Attribute	Why it matters
`trace_id` / `span_id`	The join key for evals and cost
`gen_ai.provider.name`, `gen_ai.request.model`	Cost attribution and routing analysis
`gen_ai.usage.input_tokens` / `output_tokens`	Cost computation
Prompt template ID + version hash	Regression attribution
Retrieval doc IDs + scores	Groundedness debugging
Tool name, arguments, result status	Failure and retry analysis
Tenant / user / feature identifiers	Chargeback and blast-radius
Eval scores (attached post-hoc)	Quality joins

The prompt version hash is the single highest-leverage attribute most teams omit. It is what turns "latency went up" into "latency went up because prompt v42 added three few-shot examples."

Sampling: You Cannot Afford to Keep Everything

The draft requirement — capture every prompt, every retrieved chunk, every tool response — is correct in principle and unaffordable in practice. At 10 million requests a day, full-fidelity payload capture is the dominant line item in your observability bill, often exceeding the inference cost you are trying to measure.

The answer is tiered sampling, not uniform sampling:

Always keep (100%): errors, guardrail trips, eval-score outliers, requests above a cost threshold, and any tenant on a watchlist.
Head sampling: a low baseline rate (e.g., 1–5%) for aggregate statistics.
Tail sampling: decide after the trace completes, so you can keep the interesting 1% regardless of when it occurred. This requires buffering spans until the trace closes.
Retention tiers: keep metadata (timings, token counts, model, IDs) for 90 days; keep full payloads for 7–14 days; keep a curated sample forever for eval datasets.

Separate the metrics pipeline from the payload pipeline. Metrics are cheap, high-cardinality, and always-on. Payloads are expensive, sensitive, and sampled.

Security and Privacy in Traces

Traces are the most sensitive dataset you will ever store: raw user input, retrieved proprietary documents, tool arguments that may contain credentials, and model outputs. Three non-negotiables:

Redact at the SDK boundary, not the backend. PII and secrets should never leave the process unredacted. Use attribute-level redaction rules and allowlists.
Scope access by tenant. A trace viewer that lets an engineer read any customer's prompts is a compliance incident waiting to happen. Enforce tenant-scoped encryption and access control.
Treat traces as an injection surface. A retrieved document can carry a prompt-injection payload; storing it verbatim persists the attack and can re-infect anyone reviewing the trace. Sanitize on display and flag suspicious retrieved content.

Evals — Attaching Quality to the Trace

Evals are not one thing. A credible stack runs three classes, each with a different cost and confidence profile:

Deterministic / assertion evals. Schema validity, required fields, tool-call correctness, regex and format checks. Cheap, fast, run on 100% of traffic.
Model-graded evals. An LLM judge scores groundedness, relevance, or tone against a rubric. Run on a sample (1–10%) plus 100% of flagged traces.
Human / calibrated evals. The gold standard, used to validate the judge, not to score production. A few hundred labeled examples per rubric.

The part most teams get wrong is trusting the judge without validating it. Before a model-graded eval can gate a release, measure its agreement with human labels on a held-out set (report Cohen's kappa or raw agreement). Control for known judge biases: position bias (shuffle candidate order), verbosity bias (length-confounded scoring), and self-preference (avoid judging with the same model that generated the output). An uncalibrated judge is a random number generator with good manners.

The architectural move is to write eval scores back onto the trace as attributes, not into a separate table. That is what makes questions like "which prompt version has the worst groundedness and the highest cost per resolved ticket" a single query instead of a three-way join across systems that never agreed on an ID.

Online evals close the drift gap that offline suites leave open. Sample live traffic, score it, and feed the failures back into the golden dataset on a schedule. The dataset stops being a museum piece and becomes a feedback loop.

Cost — Unit Economics, Not Invoice Reconciliation

Attributing cost to a trace is arithmetic: input_tokens × input_rate + output_tokens × output_rate, adjusted for cached tokens and reasoning tokens. The hard parts are the edges:

Reasoning / thinking tokens are billed as output on most providers and can dwarf the visible completion. Capture them explicitly.
Cached input is discounted, but the discount and cache TTL vary by provider. Model it per provider, not globally.
Batch vs. real-time pricing differs by roughly half on some providers. Tag the request mode.
Retries and failed tool calls are real spend. Amortize them into the task cost, not just the successful call.

The metric that matters to the business is not cost per token. It is cost per resolved task (or per resolved ticket, per successful agent run). Define the resolution event, attach it to the trace, and divide. That number is what lets you say "the model downgrade saved 30% per request but raised cost per resolved ticket because the resolution rate dropped" — the tradeoff that a token-level dashboard hides.

Attribution dimensions to enforce from day one: tenant, feature, team, environment, and prompt version. If a dimension is not on the span, it cannot be on the invoice.

The Reference Architecture

One pipeline, one join key:

[App / SDK] --OTel spans--> [Collector]
                               ├── metrics  --> [Metrics store]   (always-on, cheap)
                               ├── spans    --> [Trace store]     (sampled, tiered retention)
                               └── payloads --> [Payload store]   (redacted, short retention)
                                        │
                                        ▼
                              [Eval service]  --scores-->  attach to trace_id
                                        │
                                        ▼
                              [Cost engine]   --line items--> attach to trace_id
                                        │
                                        ▼
                              [Query / dashboard layer]  (joins on trace_id)

The design rules:

One trace ID is the primary key everywhere. Evals and cost are derived signals that reference it.
The collector is the control point. Sampling, redaction, and routing happen there, not in the app.
Storage is tiered. Metrics, traces, and payloads have different cost and retention profiles; do not put them in one bucket.
The query layer joins, it does not copy. Avoid ETL that duplicates payloads into a warehouse.

Build vs. Buy

Capability	Build	Buy	Recommendation
Instrumentation	OTel SDK + adapter	Vendor SDK	Build on OTel. Never let a vendor own your instrumentation.
Trace storage	Self-hosted (ClickHouse, etc.)	Managed backend	Buy below ~10M req/day; revisit at scale.
Eval execution	In-house harness	Managed eval service	Hybrid. Own the rubrics and datasets; buy the runner if it saves time.
Cost engine	Custom from token attributes	Vendor cost module	Build if multi-provider; buy if single-provider.
Dashboards	Grafana / custom	Vendor UI	Buy for speed; keep raw access for custom queries.

The rule: own the data model and the join key; rent the storage and the UI. The trace schema and the eval rubrics are your intellectual property. The dashboard is not.

30/60/90 Roadmap

Days 0–30 — Instrument and attribute.

Emit OTel GenAI spans from every LLM call path, with a pinned semantic-conventions version.
Add the mandatory attributes: trace ID, model, token counts, prompt version hash, tenant, feature.
Stand up the collector with redaction and a 1% head sample.
Ship a cost-per-trace calculation from token attributes.

**Days