The Vocabulary Budget

In the space of seven months this year, three countries launched language models with their flag on them. India’s Sarvam open-sourced two models in March, trained on compute paid for by the government’s IndiaAI Mission,1 and BharatGen, a not-for-profit based at IIT Bombay, unveiled Param 2 at the India AI Impact Summit in February.24 Portugal launched AMALIA, a model for European Portuguese, on 1 July.8 Saudi Arabia's HUMAIN announced humain-m3, a 428-billion-parameter Arabic model, on 3 September.5 Singapore’s national AI programme, meanwhile, is into the fourth generation of SEA-LION, its family of models for Southeast Asian languages.7

The pitch is the same everywhere: a model that finally speaks our language. In MyTh 002 I argued that the cheapest, least arguable unfairness in today's AI is the tokenizer tax, the extra you pay because your language is chopped into more pieces than English. A national model is supposed to be the fix. So rather than argue about it, I measured it.

What I measured

The method is the one behind the tokenizer tax interactive. Take Article 1 of the Universal Declaration of Human Rights, which has been professionally translated into almost every written language, and count how many tokens each tokenizer needs for each translation. For this entry I added seven more Indian languages to the original nineteen, and ran the lot through five tokenizers: the one behind GPT-4, the one behind GPT-4o, Google's Gemma 3, Sarvam's, and BharatGen's Param 2. To check the method, I reran OpenAI's two tokenizers the new way and compared the result with the numbers already published on this site. All 38 matched exactly.11

Three things stand out, and only one of them is the story you would expect.

The biggest drop came from California, not from India. Going from GPT-4's tokenizer to GPT-4o's cut the token count for these ten languages from 2,601 to 662, roughly three-quarters. OpenAI said as much when it launched GPT-4o in May 2024. Its own sample sentence fell from 90 tokens to 31 in Hindi, from 145 to 33 in Gujarati and from 159 to 45 in Telugu.10 Most of the tax I complained about in MyTh 002 had already been cut, in 2024, by the company the sovereign models are positioned against.

The national tokenizers do go further. Sarvam's needs about a fifth fewer tokens than GPT-4o's across the ten languages, and Param 2's about 18% fewer. Since tokens are what you pay for, wait for, and fit into a model's memory, that is a real saving.13 On Hindi, Param 2 comes within a whisker of its own English count. I do not want to undersell that.

Sarvam and Gemma 3 give the same answer on half the table. On Hindi, Marathi, Bengali, Tamil and Urdu, the counts are identical to the token. Sarvam only pulls ahead on Gujarati, Punjabi, Kannada, Malayalam and Telugu. That pattern was too neat to be chance, so I looked inside.

A vocabulary is a budget

A tokenizer's vocabulary is a fixed-size list of the pieces it is allowed to use. Gemma 3's has 262,144 entries. So does Sarvam's. When I compared the two lists entry by entry, 239,316 of Sarvam's entries (91.3%) turned out to be character-for-character identical to Gemma 3's.11

That number needs a baseline, because unrelated tokenizers overlap a lot too. They all learn the same common words and word pieces. GPT-4o's vocabulary shares about 65% of its entries with Gemma 3's, and AMALIA's about 72%.11 So 91% on its own proves nothing. What is telling is the combination: the same size to the entry, and the precise shape of the difference.

The other 22,828 entries are where it gets interesting. Classified by script, the entries only Sarvam has are almost all Indian: about 3,600 in Gurmukhi (Punjabi), 3,400 in Odia, 2,900 in Kannada, 2,600 in Gujarati, 2,500 in Telugu, 2,400 in Meetei Mayek (Manipuri), 2,100 in Malayalam and 1,300 in Ol Chiki (Santali). The same number of entries that only Gemma 3 has are mostly Korean, Japanese, Latin-script, Thai, Chinese and Ethiopic.11 The simplest reading is that the vocabulary was kept at the same size and around 23,000 slots were reassigned, mostly away from East and Southeast Asian scripts and towards Indian ones. That fits exactly with what Sarvam itself says its tokenizer is for: it singles out Odia, Santali and Manipuri as the languages where it is “significantly more efficient”.2 It also explains the table. Devanagari, Bengali and Tamil script, and the Arabic script Urdu uses, were already well covered in Gemma 3, so nothing changed for them.

Building on a strong existing vocabulary and adapting it is sensible engineering, and nothing in Sarvam's own account rules it out. Its blog says the models were trained from scratch and that it “optimized tokenization”, and both can be true at once.2 My point is a different one. A fixed-size vocabulary is a zero-sum budget, so a slot spent on Kannada is a slot not spent on something else. You can see who paid for this one. On the same UDHR sentence, Sarvam's tokenizer needs 424 tokens for Thai against Gemma 3's 62, 219 for Korean against 49, 183 for Japanese against 46, and 309 for Amharic against 83.11

For a model built to serve India, that is a defensible choice. Nobody should be running Thai through Sarvam. But it punctures the idea that the tokenizer tax gets solved one national model at a time. Each national vocabulary moves the tax onto the speakers it was not built for. Of the tokenizers I measured, the one that did best on Amharic was Google’s Gemma 3, at 83 tokens against 206 for GPT-4o and 305 for Param 2. It was not anyone’s national one. Speakers of languages without a well-funded state behind them depend, as before, on whether the big general-purpose models decide to include them.

Whose foundations?

“Sovereign” is doing a lot of work in these launches, so it is worth itemising what each one actually owns. Every claim below is from the builders' own announcements or model cards, except the tokenizer figures, which are mine.

None of this is a scandal. Fine-tuning an open model is how most of the field works, and choosing a strong partner is a reasonable way to move fast. But it does mean “sovereign” covers very different things. It can mean training from scratch on home compute. It can mean commissioning a model from a foreign company under that company's licence. It can mean post-training someone else's model on local data. These are different arrangements, with different consequences for who can change the model, who can withdraw it, and who learns how to build the next one. They deserve different words.

A nation is not a language

The deeper problem with the national framing is that the unit is wrong. India alone has 22 scheduled languages written in 12 scripts, which is exactly why Sarvam had to make the choices it did.2 “An Indian model” is a coalition of many language models sharing one budget.

Arabic shows the same problem from the other direction. humain-m3 is described as an Arabic model trained on Arabic-native content.5 But the Arabic people speak varies a great deal from country to country. In our own work we tested how large language models, mainly GPT-4, handle health claims written in Saudi, Egyptian, Lebanese and Moroccan Arabic. They were good at recognising true claims and struggled with mixed and ambiguous ones, “especially in underrepresented dialects”.12 A single “Arabic” score on a benchmark tells you little about which Arabic.

Then there is the detail in Sarvam's launch post that I keep coming back to. Describing how its Indus assistant answers a question asked in an Indian language, the post says: “Web search is conducted in English to access current and comprehensive information, while the model interprets the query and delivers a correct response”.2 That is an honest and sensible engineering choice. It is also the point from MyTh 002, in one sentence: what decides who a model serves is where the written-down knowledge is, not how many people speak the language. You can build the model at home, train it on home compute and give it a tokenizer that respects your scripts. When it needs to know something current, it still goes and reads it in English.

And the scores are mostly self-reported

One last caution, which applies to every launch above. Sarvam's benchmark tables are its own.2 HUMAIN's claim that humain-m3 achieved “the highest average score” across seven Arabic benchmarks comes from its own evaluation.5 That is normal for a launch, and it is not an accusation. But MyTh 002 described work showing that cultural evaluations of language models can move more when the question is reformatted than when the country is changed.14 Until there are independent evaluations, in the actual dialects and registers people use, claims that a national model “understands” its country should be read as hopes.

Build them, but stop calling them a fix

I am in favour of these models. Someone has to care about Odia, Santali and Manipuri tokenization, and the market on its own was never going to. Sarvam making tokenizer efficiency a headline feature, and publishing a comparison with other tokenizers, is the easy win MyTh 002 complained nobody was taking. Param 2 bringing Hindi close to its own English cost is a real achievement. But four things follow from the measurements, and none of them is in the launch speeches.

• Itemise sovereignty. Data, tokenizer, weights, compute and licence are five separate questions, and the answer can be “ours” for some and “someone else's” for others. Any government buying or funding a “sovereign” model should ask all five and publish the answers.
• Treat the vocabulary as public policy. If a national tokenizer saves your languages money by charging other languages more, that is a reasonable trade, but it should be a stated one. Publish token cost per language, including the languages you made more expensive.
• The tax is not only a national problem. The speakers who pay the most, such as Amharic in my measurements, are the ones least likely to get a well-funded national model. The general-purpose models still matter most to them, and pressure on those models' tokenizers is still worth applying. It worked once already: 2,601 tokens down to 662.
• The lasting asset is the writing, not the weights. Model weights date quickly. A corpus of well-curated, machine-readable text in your languages will not, and it is the only thing that changes where a model has to go to look things up. If I were advising a ministry, that is where I would put the money.

The national model is a good idea sold with the wrong slogan. It does not make the language problem go away. It moves it, prices it and hands it to someone. The least we can ask is to be told who.