This page reproduces the token counts and vocabulary comparison in MyTh 009, The Vocabulary Budget. It downloads six public tokenizers from Hugging Face, tokenizes Article 1 of the Universal Declaration of Human Rights in 26 languages, and compares each vocabulary with Gemma 3's. The output below is the data published as tokenizers.json. It takes a minute or two, depending on your connection.
Running…