LLM Evaluation in Production: Building the Evals Pipeline Your ML Team Can Actually Trust
How to build an LLM eval pipeline your team can actually trust in production — from golden sets to regression gates.
Expert Q&A
Q: How do we set automated quality thresholds without them being arbitrary or causing alert fatigue? A: Threshold-setting is more art than science, but you can ground it in data. Start by establishing a baseline pass rate from your current production outputs—sample 100-200 real user interactions and score them manually. This gives you a defensible starting point. For critical test cases (safety, compliance, core functionality), set thresholds at 95-99% pass rate. For quality dimensions like helpfulness or tone, allow 80-85% to account for subjectivity. Implement graduated alerting: warn at 90% of threshold, block at threshold breach. This prevents alert fatigue while catching drift early. Review and adjust thresholds quarterly based on user complaint rates—if users aren't complaining at your threshold, it's calibrated correctly.
Q: Our eval costs are spiraling. How do we balance thorough evaluation with budget constraints? A: Stratified sampling is your primary cost lever. Not all test cases deserve equal scrutiny. Tier your test suite: run 100% of critical path tests on every commit, 20% of edge case tests daily, and full regression suites weekly. Use smart sampling based on code change analysis—if a prompt modification affects customer support responses, prioritize that test subset. Cache aggressively—many eval runs test the same inputs against slightly modified prompts; memoize results where outputs haven't changed. For LLM-as-judge evaluations, consider using smaller, cheaper models (GPT-3.5-class) for routine checks, reserving GPT-4-class models for disputed cases only. Our teams have reduced eval costs 60-70% through tiered execution without meaningfully reducing defect detection.
Q: What's the minimum viable eval pipeline for a team of 2-3 engineers just starting out? A: Start with three components, no more: First, a version-controlled test suite with 50-100 representative input-output pairs across your core use cases. Store this alongside your prompt code. Second, a pre-commit hook that runs your test suite on every prompt change before code reaches version control. Third, a simple Slack alert that posts pass/fail results to your team channel. This gives you versioning, automated gates, and visibility without infrastructure overhead. As you scale, layer in scheduled regression runs, then dashboards, then online evaluation. Don't build the mansion before you need it.
Q: How do we evaluate qualities that are inherently subjective—like whether a response is "helpful" or has the right tone? A: Human evaluation remains the gold standard for subjectivity, but you can systematize it. Create explicit rubrics that define "helpful" into observable criteria: does it answer the user's question? does it provide relevant context? does it avoid unnecessary verbosity? Train your team on rubric application and measure inter-rater reliability (aim for 80%+ agreement before scaling). Use LLM-as-judge for catch-all scoring, but validate it against human labels regularly—LLM judges have their own biases. For tone evaluation, consider classifier-based approaches: train a small classifier on human-labeled examples of "appropriate tone" vs. "too aggressive" or "too casual." This gives you faster, more consistent scoring than pure human judgment for routine evaluation.
Q: Our reference answers keep becoming outdated as our product evolves. How do we handle eval drift? A: Eval drift is a signal your evaluation strategy needs restructuring. Move from static reference answers to reference principles. Instead of "the correct response to 'How do I cancel?' is X," define evaluation criteria like "response includes cancellation steps, mentions refund policy, provides confirmation mechanism." This creates durable rubrics that survive product changes. Implement regression testing against known-bad outputs: maintain a collection of inputs that previously produced failures and ensure they don't regress. For rapidly evolving features, use delta evaluation: instead of scoring absolute quality, score whether a new version is better or worse than the previous version on the same inputs. This sidesteps the need for perfect reference answers entirely.
Q: How do we integrate eval pipelines into CI/CD without creating deployment bottlenecks that frustrate our engineering team? A: Speed and coverage are in tension—manage it through parallelization and selective execution. First, ensure your eval engine runs tests in parallel; a 10-minute serial test suite becomes 2 minutes with 5 parallel workers. Second, implement canary evaluation: deploy to 5% of traffic, run focused eval suite against production traffic, promote or rollback based on results. Third, gate only critical paths in the main CI pipeline; run comprehensive suites asynchronously post-deploy. The key is making eval feedback fast enough to not interrupt flow state—our teams target <3 minutes for gated evaluation. If your eval suite takes longer, it's a bottleneck that will be bypassed.
Illustration Review
The existing architecture diagram block is appropriately placed and well-scoped. However, the following concepts would benefit from visual explanation:
--> -->