[Paper Review] Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models

Orevaoghene Ahia, Sachin Kumar+5 more|arXiv (Cornell University)|May 23, 2023

Taxation and Compliance StudiesEconomics, Econometrics and Finance3 citations

TL;DR

This paper investigates how subword tokenization in commercial language model APIs leads to disproportionate token counts and higher costs for non-Latin script languages, despite equivalent semantic content. It demonstrates that speakers of underrepresented languages face higher financial costs and reduced model performance due to inherent linguistic and encoding biases in tokenization, urging vendors to adopt fairer pricing and transparency.

ABSTRACT

Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based on usage, more specifically on the number of ``tokens'' processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. In this work, we analyze the effect of this non-uniformity on the fairness of an API's pricing policy across languages. We conduct a systematic analysis of the cost and utility of OpenAI's language model API on multilingual benchmarks in 22 typologically diverse languages. We show evidence that speakers of a large number of the supported languages are overcharged while obtaining poorer results. These speakers tend to also come from regions where the APIs are less affordable to begin with. Through these analyses, we aim to increase transparency around language model APIs' pricing policies and encourage the vendors to make them more equitable.

Motivation & Objective

• To investigate whether tokenization in commercial language model APIs leads to unequal costs across languages due to subword fragmentation.
• To examine how tokenization disparities affect both the economic cost and model utility for different languages.
• To assess whether these disparities exacerbate existing socio-economic inequities in access to NLP technologies.
• To highlight the role of tokenizers as a critical, often overlooked component in the fairness of commercial LLMs.
• To advocate for greater transparency and equity in API pricing and model design for multilingual users.

Proposed method

• Used OpenAI's API and tokenizer to process parallel text from the FLORES-200 benchmark across 22 typologically diverse languages.
• Measured tokenization fragmentation rates by comparing input text length (in characters) to output token count across languages.
• Quantified cost disparities by calculating the number of tokens required to process the same semantic content in different languages.
• Evaluated model utility using in-context learning performance on multilingual benchmarks, correlating it with token count and language type.
• Analyzed the correlation between high token cost, poor performance, and socio-economic factors such as regional purchasing power.
• Used CC100 as a proxy for training data distribution to infer language representation biases in the model's pretraining data.

Figure 1: We investigate the effects of subword tokenization in LLMs across languages with different writing systems. Our findings highlight disparities in the utility of LLMs, as well as socio-economic disparities and increased costs in using commercial APIs for speakers of underrepresented languag

Experimental results

Research questions

• RQ1 How does subword tokenization vary across languages with different writing systems, and what is the resulting impact on token count?
• RQ2 To what extent do tokenization differences lead to disproportionate API costs for speakers of underrepresented languages?
• RQ3 Does higher tokenization fragmentation correlate with reduced model utility in in-context learning tasks?
• RQ4 Are the languages with higher costs and lower performance also those spoken in regions with lower average purchasing power for API usage?
• RQ5 What role do linguistic and encoding properties (e.g., script type, Unicode representation) play in tokenization bias?

Key findings

• Languages with non-Latin scripts, particularly those using CJK, Arabic, and other complex writing systems, experience significantly higher tokenization fragmentation than Latin-based languages.
• The same semantic content required up to 2.5x more tokens in languages like Japanese and Chinese compared to English, directly increasing API costs.
• Languages with higher token counts showed lower performance in in-context learning benchmarks, indicating reduced model utility despite identical input content.
• Speakers of over-fragmented languages are often from regions with lower average income, making the higher API costs more burdensome and exacerbating digital inequity.
• The cost disparity is not solely due to data scarcity but is rooted in inherent linguistic and encoding characteristics, such as Unicode script complexity and character frequency patterns.
• The study reveals that current API pricing models based on token count are fundamentally unfair to non-Latin script users, despite the same information being conveyed.

Figure 2: Average number of tokens by script after tokenizing the Flores dataset. The fragmentation rate is lower for Latin script languages and higher for other scripts. Number of languages per language group is indicated at the top of each bar.