LLMOps Tooling in 2026: Building the Enterprise AI Stack That Survives Production

Build, Buy, or Compose: The 2026 Decision Framework

Intro ~70:

"In 2024 the question was 'which chatbot wrapper?' In 2026 it is which seams do you own. Every layer above can be bought, built, or composed. The right answer depends on three variables: team size, regulatory exposure, and how differentiating each capability actually is. Choose per layer, not per vendor — the stack is a portfolio, not a marriage."

~60. Good.

When a consolidated platform wins (~165)

"A consolidated platform bundles evaluation, observability, prompt management, and sometimes gatewaying under one roof.

Choose consolidation when AI is important but not your core product. If three to eight engineers ship AI features inside a larger product, integration time is your scarcest resource, and one vendor with one data model beats four vendors with four SDKs. Unified data is the underrated benefit: when traces, evals, and prompts share IDs, 'this regression came from Tuesday's prompt change' is a query, not an investigation.

Consolidation also wins early: pre-PMF teams need speed more than leverage, and a single support channel beats a vendor blame triangle at 2 a.m.

The costs are real, though. Platform pricing scales faster than point tools, roadmap changes arrive on someone else's schedule, and deep customization — custom judges, exotic data pipelines — hits platform ceilings. Lock-in risk concentrates where your data model lives; mitigate it by insisting on export from day one."

~160. Good.

When best-of-breed composition wins (~165)

"Composition — a gateway here, an eval platform there, your existing observability estate underneath — wins under three conditions.

First, regulated industries with hard residency requirements: banks and healthcare systems routinely need self-hosted or VPC-deployed components, and few consolidated platforms cover every layer in-region. Composition lets you place each capability where compliance demands.

Second, teams with an existing observability estate. If Datadog or Grafana already carries your on-call culture, adopt the OpenTelemetry GenAI conventions and forward traces there rather than teaching a second pane of glass.

Third, scale. At meaningful volume, point tools price better, and multi-vendor contracts are negotiating leverage.

The honest cost is integration tax: four vendors mean four SDKs, four upgrade cycles, and four support queues, plus version drift between them. Budget one engineer-month per integration per year, at minimum. Composition is the right default for platform teams; it is an expensive hobby for product teams."

~160. Good.

The TCO model (~170)

"Model three-year total cost of ownership across five line items, not license fees alone.

1. Licenses and usage: seats, traced spans, eval runs — pricing models differ wildly; model your actual trace volume.
2. Infrastructure: GPUs if self-hosting, egress, storage for traces.
3. Integration engineering: the line item everyone forgets; each tool costs integration time at purchase and at every major upgrade.
4. Quality maintenance: golden datasets rot; budget quarterly refresh cycles.
5. Incident and exit costs: what does a provider outage cost per hour, and what does switching cost per layer?

Self-hosted open-weight serving can run 2–5x cheaper than frontier APIs at sustained high utilization — but below roughly 30–40% GPU utilization, the API usually wins; model your traffic before believing the multiple. (modeled range; validate against your traffic)

Run the model per layer. Most 2026 stacks converge on a hybrid: buy evaluation and observability, compose the gateway, and self-host only high-volume, latency-sensitive, or residency-bound workloads. If a vendor cannot tell you their exit path in one sentence, that is your answer."

~180. Slightly over; fine.

The 2026 Vendor Landscape by Category

Intro ~50:

"The market consolidated in 2025 and then fragmented again along agent lines. Evaluation and observability are merging; agent tooling is exploding. Names below are representative, not exhaustive — and inclusion is not endorsement. Categories matter more than logos; capabilities shift quarterly."

~45. Good.

Evaluation & observability platforms (~200)

"This is the most contested layer, and the lines are blurring: every serious observability vendor added evals; every eval vendor added tracing.

LangSmith (LangChain) offers the tightest loop if you build on LangChain and LangGraph: prompt management, datasets, CI integration, and agent monitoring in one pane. Braintrust leads on developer experience — its eval loops and playgrounds feel like a proper test framework. Arize Phoenix is the open-source favorite for teams that want self-hosting and OTel-native tracing. Langfuse, also open source, pairs prompt management with usage-based pricing that finance tends to like. W&B Weave extends the Weights & Biases experiment-tracking pedigree into LLM evals; Galileo focuses on production hallucination detection.

Selection guidance: pick the one that fits your CI, not your demo. Evaluation value comes from running thousands of times, so SDK quality and CI ergonomics beat dashboard beauty. Insist on OTel export regardless of choice."

~170. Slightly under 200; fine.

AI gateways & routers (~150)

"Gateways became boring infrastructure in the best sense. LiteLLM dominates the open-source self-hosted path with broad provider coverage and a proxy server that finance can query. Portkey adds guardrails and caching on top of routing; Helicone pairs gatewaying with observability. Kong and Cloudflare push AI gateways into existing edge and API-management estates, which shortens security reviews.

Evaluate on: protocol coverage (including embeddings and batch APIs), fallback and retry logic, budget enforcement granularity, and whether it self-hosts. The gateway is your chokepoint of leverage — it is the one layer where switching costs should approach zero, so keep it stateless and boring."

~130. OK.

Agent observability: fastest-growing subcategory (~150)

"Agents broke flat request logs, and a tooling subcategory grew to fix it. Agent observability tools reconstruct decision trees: every plan, tool call, retry, and handoff, stitched into session-level narratives.

LangSmith and Langfuse added agent views; Arize and Galileo added trajectory scoring; AgentOps and similar entrants focus purely on agent runs. The differentiators are trajectory evaluation — scoring the path, not just the answer — and loop detection that catches agents before they burn budgets.

Adopt the OpenTelemetry GenAI conventions even if you buy: they keep agent traces portable across vendors, and they make your agent telemetry a first-class citizen in existing APM. This subcategory will consolidate; portable telemetry is your hedge."

~130.

Guardrails & security tooling (~150)

"Security tooling split into three clusters. Policy engines — NeMo Guardrails, Guardrails AI — let teams define input/output rules as code. Specialized threat vendors — Lakera, Prompt Security — focus on injection detection, jailbreaks, and red-teaming, with threat intelligence the generalists lack. Model-level classifiers such as Meta's Llama Guard and cloud content-safety APIs provide cheap baseline filtering.

For agentic systems, prioritize tool-call filtering and egress control: the injection that matters targets your tools, not your chat window. Demand red-team evidence, not marketing claims — ask how the vendor benchmarks against the OWASP LLM Top 10 and what their false-positive rates look like on your traffic."

~120.

Inference platforms & self-hosted serving (~150)

"Self-hosting matured from hobby to procurement category. vLLM is the default open-source serving runtime, with TensorRT-LLM competing on raw NVIDIA throughput. Ray Serve and BentoML handle orchestration and packaging; Modal and RunPod rent you GPUs without capacity commitments.

Managed inference platforms — Fireworks, Together, Groq — offer open-weight models with API convenience and better unit economics than frontier APIs for suitable workloads.

The 2026 pattern: frontier APIs for hard reasoning, open-weight models for volume. Route classification, extraction, and summarization to cheaper tiers; reserve frontier models for planning and generation. Hyperscalers (Bedrock, Vertex, Azure AI) win when residency and existing commitments dominate. Choose by latency SLO, residency, and utilization — in that order."

~135.

Selection scorecard: 12 criteria (~150)

"Score every candidate 1–5 on these twelve; anything below 3 on items 1, 4, or 12 is disqualifying.

1. OpenTelemetry-native trace export
2. Self-host or VPC deployment option
3. Data residency and region pinning
4. Complete data export — no hostage formats
5. SSO, RBAC, SCIM
6. Prompt registry with versioning and rollback
7. CI-native evaluation hooks
8. Golden dataset management
9. Agent trajectory support
10. Per-team cost attribution
11. Pricing predictability at 10x volume
12. Documented exit path

Weight criteria by layer, not uniformly — an eval tool lives or dies on CI ergonomics; a gateway on protocol coverage. Re-score annually; this market moves quarterly."

~120. Good.

The 90-Day Production Playbook

Intro sentence or two? The outline has illustration then three H3s. Maybe one intro line: "Ninety days from uninstrumented to audit-ready, assuming two to three engineers." Then sections.

Days 0–30: Instrument everything (~200)

"Resist the urge to buy anything strategic yet; instrument first.

Week 1: deploy an AI gateway in front of every model call. Route all traffic through the gateway from day one, even if you use one provider — retrofitting chokepoints after features ship is politically brutal. Turn on per-team, per-feature token accounting.

Weeks 2–3: add tracing with OpenTelemetry GenAI conventions. Capture 100% of errors and 10–20% of successes; alert on trace completeness, not just errors. Cost attribution lands this week too: every trace tagged with team, feature, and environment, so finance sees unit costs, not one invoice.

Week 4: build the first golden dataset — 50 to 200 real inputs with expert-validated outputs, drawn from support tickets and production logs, not invented. Baseline your current quality before you change anything, because 'before' numbers are unrecoverable later.

Exit criteria: every model call traced and attributed, one dashboard finance trusts, one golden set versioned in Git."

~170.

Days 31–60: Evaluation harness in CI (~200)

"Month two turns measurement into a gate.

Weeks 5–6: calibrate an LLM-as-judge against your golden set — measure agreement with human labels, target 85–90%+ before trusting it, and keep humans in the loop for disagreements. Add the RAG triad if you retrieve: context relevance, groundedness, answer relevance. Calibration is the whole game; an uncalibrated judge automates your blind spots.

Weeks 7–8: wire evaluation into CI. A regression gate blocks deploys when quality drops below threshold, exactly like failing unit tests. Run evals on pull requests that touch prompts, retrieval configs, or model versions. Add a nightly run against the full golden set, since providers update models without asking.

Also ship