Search papers, labs, and topics across Lattice.
This study introduces the Tokenization Equity Audit (TEA), a benchmark for assessing the economic impact of tokenization disparities across languages in large language models. By analyzing a Python debugging corpus translated into Bengali, Hindi, Arabic, Tamil, and Yoruba, the research reveals that Bengali requires 1.56 times more tokens than English when using the GPT-4o tokenizer, while Yoruba shows an even higher premium of 2.37 times. These findings underscore the significant barriers that tokenization inequities impose on underserved language communities, particularly in educational contexts reliant on AI tools.
Tokenization disparities can inflate costs and reduce effective context length for underserved languages, with Bengali and Yoruba facing tokenization premiums of up to 4.5 times that of English.
Large language models are increasingly deployed as general-purpose educational and technical assistance systems, but their underlying infrastructure does not treat languages equally. One underexamined source of disparity is tokenization: semantically equivalent content can require substantially different token counts across languages, affecting API cost, latency, and usable context length before a model is invoked. We introduce the Tokenization Equity Audit (TEA), a reproducible benchmark for measuring tokenization premiums in technical tutoring content. TEA evaluates three widely used tokenizers, GPT-4o's o200k base, Qwen2.5-7B, and Mistral-7B, on a 120-item Python debugging corpus translated from English into Bengali, Hindi, Arabic, Tamil, and Yoruba. Bengali and Hindi serve as the primary validated cases, while the remaining languages provide exploratory cross-script and cross-family comparisons. Across this corpus, Bengali requires (1.56\times) as many GPT-4o tokens as English, reducing a nominal 128k-token context window to an effective 82k-token English-equivalent capacity for the same semantic content. With the Qwen2.5 and Mistral tokenizers, Bengali requires up to (4.5\times) the English token count. Yoruba, despite using the Latin script, exhibits the highest GPT-4o tokenization premium at (2.37\times), indicating that tokenization inequity cannot be explained by script family alone. These results demonstrate that tokenization can create measurable economic and functional barriers, highlighting the need to treat tokenization as an equity-relevant infrastructure layer for underserved language communities, particularly where educational systems depend on low-cost or offline-capable AI tools.
