The researchers describe this as a hidden “tokenizer tax” and warn that it exacerbates digital inequality.
This disparity is not limited to languages with few speakers, the researchers note. “If we sort by speakers instead of web presence, the tax lands on enormous populations: Punjabi (~200 million speakers), Bengali (~266 million), Urdu (~288 million), Burmese (~36 million), Amharic (~36 million). This means hundreds of millions of people billed at multiples of the English rate for the same intelligence,” the study says.
Languages that use writing systems not as commonly found in the AI models’ training data are particularly affected. Examples include Burmese, Tibetan, Khmer, and Lao. Languages that use the Latin alphabet, on the other hand, often have an advantage, even when spoken by relatively few people.