NLP & LLMsmultilingual LLM performanceLLM evaluationtokenizationmultilingual benchmarks

The Multilingual Quality Gap: Why LLMs Still Underperform Outside English

Why LLMs underperform in non-English languages: tokenization, benchmark bias, and data gaps—and what B2B teams can do to close the multilingual quality gap.

What It Is and Why It Persists

A model that scores near the top of an English reasoning benchmark can degrade sharply the moment you ask it to do the same work in Bengali, Swahili, or Quechua. This is not a hypothetical edge case. On multilingual evaluation suites, the spread between top-tier English performance and low-resource-language performance is frequently double-digit percentage points—and the gap widens further on tasks that require cultural context, idiom, or register. The exact size of the gap varies by language, task, and model family; treat any single headline figure with suspicion.

For B2B teams, the stakes are concrete. A global product rollout built on English-only quality assumptions ships defects into every non-English market. Support automation misfires on the languages your customers actually speak. And unreviewed machine output leaks into adjacent workflows—internal communications, draft localization, customer-facing support—where legal and regulatory teams will eventually find it. (Certified translation pipelines in regulated domains typically mandate human review; the risk concentrates in the workflows around them.)

The convenient explanation is that low-resource languages simply lack training data. That story is incomplete. The multilingual quality gap is structural: tokenization, benchmark design, evaluation methodology, and data economics compound one another. Fixing it requires understanding each layer, not just pouring more tokens into the pipeline. (See our related guide on LLM evaluation best practices for enterprise teams.)

What the Multilingual Quality Gap Actually Looks Like

It helps to separate two failure modes that get conflated in vendor claims and internal reporting.

A capability gap means the model cannot perform the task at all—it produces incoherent output, hallucinates structure, or refuses. A quality gap means the model performs the task, but worse and less consistently than it does in English. The second is more dangerous in production because it fails silently. Output looks plausible, passes a casual review, and ships.

Where Multilingual LLM Performance Drops

Degradation is not uniform across tasks. Translation and summarization tend to hold up better than open-ended reasoning, because they anchor the model to source text. Reasoning tasks—multi-step math, instruction following, structured extraction—degrade more sharply, and the degradation is often nonlinear: a model may handle a simple query in Hindi but fail a compound one that it would breeze through in English.

The pattern holds across model families and vendors. Scale helps at the margin but does not close the gap. A larger model trained predominantly on English still carries English-shaped priors into every other language it touches.

Why Averages Hide the Problem

Aggregate multilingual scores are the single most misleading metric in enterprise AI procurement. A benchmark that reports an "87% average across 20 languages" (illustrative) may be built on Spanish, German, French, and Japanese scores near 92%—pulling up a long tail of languages scoring in the 50s and 60s.

This is aggregation masking, and it is endemic. High-resource languages dominate both the benchmark composition and the reported headline number. The languages where your customers actually live—Indonesian, Yoruba, Tamil, Polish, Vietnamese—often sit in the tail that the average conceals.

If your multilingual evaluation reports a single number, you do not have a multilingual evaluation. You have an English evaluation with decorative footnotes.

Buyers should demand per-language breakdowns, weighted by their actual traffic distribution, before accepting any quality claim. A useful procurement rule: reject any vendor scorecard that does not show the per-language sample size alongside the per-language score. (More on this in our breakdown of how to benchmark multilingual LLM quality.)

Root Cause 1 — Tokenization Penalizes Non-Latin Scripts

Every major LLM converts text into subword tokens before processing. The tokenizer is trained on a corpus, and that corpus is overwhelmingly English. The result is a systematic penalty for scripts and languages that were underrepresented during tokenizer construction.

The Tokenizer Fertility Problem

Researchers measure this with fertility: the average number of tokens a tokenizer produces per word. For common English vocabulary, modern tokenizers (e.g., the o200k and Llama 3 families) land near 1.0–1.3 tokens per word. Languages written in Devanagari, Arabic, Bengali, Amharic, or Thai frequently require 1.5 to over 3 times as many tokens to represent equivalent content, depending on the tokenizer and the script's Unicode properties. Languages with rich morphology like Finnish or Turkish, and scripts without whitespace delimiters, suffer similarly.

This is not a minor efficiency detail. It is a structural cost applied to every non-English request, baked in before the model performs a single inference step. The exact ratios vary substantially by tokenizer and language, so treat published figures as directional—measure fertility on your own content rather than trusting a vendor's table.

Downstream Effects on Cost and Context

The fertility penalty compounds across three dimensions:

  • Inference cost. More tokens per document means more compute per request. A support ticket in Bengali can cost two to three times what the same ticket costs in English.
  • Context window consumption. A document that fits comfortably in a 128K-token window in English may truncate in a high-fertility language, silently dropping content the model needed.
  • Weaker cross-lingual transfer. Fragmented tokenization splits semantically related words into unrelated fragments, which appears to contribute to weaker generalization of concepts learned in one language to another. The mechanism is an active area of research rather than a settled causal claim, but the correlation is consistent across model families.

Side-by-side tokenization of the same sentence in English, Hindi, and Swahili, showing token counts
Side-by-side tokenization of the same sentence in English, Hindi, and Swahili, showing token counts

None of these effects appear in a headline benchmark score. All of them appear in your production bill and your error logs. (Related: reducing LLM inference costs in production.)

Root Cause 2 — Benchmark Evaluation Was Built for English

The benchmarks used to certify multilingual quality were largely designed in English and adapted afterward. That adaptation process introduces artifacts that inflate scores and obscure real weaknesses.

Translation Artifacts and Cultural Mismatch

Many multilingual benchmarks are produced by translating English test items into target languages. Machine-translated or hastily human-translated items carry predictable defects: unnatural phrasing that signals "this is a test," loss of idiom, and cultural referents (holidays, institutions, units, legal concepts) that do not map cleanly. A model can score well by pattern-matching the translationese rather than by demonstrating competence.

Saturation and Contamination

Popular English benchmarks (MMLU, GSM8K, and their descendants) are heavily saturated and increasingly contaminated in training data. Their multilingual derivatives inherit both problems. Newer suites—Belebele, FLORES-200, Global MMLU, MMMLU—were designed with multilingual coverage in mind and are better anchors, but none is a substitute for evaluation on your own domain data in your own languages.

Root Cause 3 — Data Economics and the Long Tail

The data story is real, just not the whole story. Pretraining corpora are dominated by English and a handful of high-resource languages; Common Crawl's language distribution is heavily skewed, and curated multilingual data is expensive to source, license, and annotate. The economics are unforgiving: annotation cost per token rises sharply for low-resource languages because qualified annotators are scarcer, and quality control is harder to automate.

Two consequences matter for buyers:

  • Instruction-tuning and RLHF coverage is uneven. A model may have strong multilingual pretraining coverage but thin multilingual alignment data, which shows up as poor instruction following and inconsistent formatting in the tail languages—even when raw comprehension is fine.
  • The tail is long and moving. "Low-resource" is not a fixed list. Languages with growing digital footprints (e.g., Indonesian, Vietnamese) are better served than they were two years ago; others remain underserved. Vendor claims age quickly.

Root Cause 4 — Evaluation Methodology

Even with good benchmarks, enterprise evaluation methodology often undermines the result. Three recurring failures:

  • English-source evaluation only. Testing a multilingual system by translating English prompts and grading against English references misses register, formality, and locale-specific correctness.
  • LLM-as-judge without calibration. Using a strong English-centric model as an automatic judge introduces its own multilingual bias; judges must be validated against human ratings per language.
  • No production-distribution weighting. A 95% score on a language that is 0.5% of your traffic is not the number that matters. Weight per-language scores by actual request volume and error cost.

What to Do About It

The gap is structural, but it is measurable and manageable. A practical sequence for B2B teams:

  1. Measure fertility on your own corpus per target language and tokenizer; use it to forecast cost and context-window risk before rollout.
  2. Demand per-language scorecards with sample sizes, weighted by your traffic distribution—not a single average.
  3. Evaluate on your domain data, not just public benchmarks, and validate any LLM-as-judge against human raters per language.
  4. Separate capability from quality in acceptance criteria; set explicit per-language quality thresholds rather than a global pass/fail.
  5. Budget for the tail. Low-resource languages will cost more per request and require more human review; plan for it rather than discovering it in production.

The multilingual quality gap will not close on its own, and a bigger English-trained model will not close it for you. Teams that treat it as an engineering and procurement problem—measured per language, weighted by real traffic, validated against real users—ship fewer silent failures into the markets they are trying to win.

ShareX / TwitterLinkedIn
← Back to Research