The Tokenizer Tax: Some Languages Cost Up to 17 Times More Tokens in the Same AI Model

People who use less widely spoken languages may pay more to use the same AI model than English speakers do — and a new study has put hard numbers on the gap. The Coalition for Language Equity in AI Research (CLEAR), in work reported by Rest of World on October 27, calls it the "tokenizer tax": because tokenizers split less-represented languages into more pieces, the same sentence can cost several times more in tokens depending on the language it is written in.

Among the 20 most popular languages on the internet, the most expensive used between 1.5x and 5x more tokens than the cheapest, with most models clustered between 1.5x and 3.1x. Expand the analysis to 200 languages and the spread blows out: across FLORES-200, a professionally translated benchmark covering about 200 languages, CLEAR measured max/min token ratios between 2.7x and 17x across 19 model families.

CLEAR's own example uses one sentence — "People should benefit from AI, regardless of the language they speak." With GPT-5's tokenizer it counts 13 tokens in English, 26 in Malayalam and 78 in Amharic, where each Ge'ez character splits into two single-byte tokens, making Amharic input six times more expensive. The study also pre-empts the obvious objection that some languages are simply wordier: the most verbose language in its dataset, Shan, is only 42 percent longer than English at the character level, an effect measured "in tens of percent" that cannot explain a 17x gap.

On CLEAR's grading scale — a ratio of 3x or less earns an A, up to 6x a B, up to 9x a C, up to 12x a D, anything above 12x an F — no general-purpose frontier model scores better than a B. Google's Gemini and Gemma family leads at 5.1x (grade B). Anthropic's Claude Fable 5 and Opus 4.8 sit at 9.3x and Meta's Llama 4 Scout and Muse Glimmer at 9.5x, both grade D, with DeepSeek's V4.1 Flash at 10.6x. GPT-5 and gpt-oss grade F at 13.8x, as do Mistral's Ministral 3 and NVIDIA's Nemotron 3.5 at 14.9x each and TII's Falcon 3 at 17.0x. Meta's translation-only NLLB-200 tops the table at 2.7x — proof, CLEAR argues, that a vocabulary built for many languages works, though it is not a fair bar for a frontier model that must also handle code and math.

The counterintuitive finding is that script matters more than speaker count. Asturian, which has relatively few speakers and no Common Crawl presence, benefits from the Latin-script advantage and overlap with related Romance languages. The steepest penalties fall consistently on languages written in scripts tokenizers rarely saw during training — Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki — across nearly every model analyzed.

The tax extends beyond billing. Higher token counts mean higher latency and a smaller effective context window, so conversation history and documents crowd out useful context faster. "Think about a tutor: retaining the student's earlier mistakes and explanations is part of delivering a good lesson," said Pratyush Choudhury, co-founder of Activate AI, India's first venture fund dedicated exclusively to AI. "If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service."

Better training data alone will not fix it, the study's critics and commenters agree. Raoul Nanavati, co-founder and CEO of Mumbai-based Navana.ai, pointed to RESPIN, a Gates Foundation-funded initiative with the Indian Institute of Science that collected text and read speech from native speakers of 38 dialects. But Dheemanth Reddy, CEO of Maya Research, argued that falling token prices and better data still leave the structural problem intact: labs will have to deliberately design future tokenizers for diverse scripts, and treat language efficiency as a product requirement alongside accuracy and speed. Until then, the tokenizer remains an English-centric artifact — and a silent surcharge on the next five billion users.