Interactive · Measured on this page

The Tokenizer Tax

Language models do not read characters. They read tokens, and the tokenizer that produces them was fitted to whatever the model saw most of — which was English. So the same sentence, saying the same thing, costs a different amount depending on the language it happens to be written in. Tokens are the unit of billing, the unit of latency, and the unit of memory. Here is what that costs, measured.

—Most expensive language here, against English, on GPT‑4's tokenizer
—Languages measured, spanning — writing systems
—Tokens for the English passage — the baseline everything is divided by
—Languages where the newer tokenizer cut the tax by half or more

One sentence, nineteen languages

Article 1 of the Universal Declaration of Human Rights — the same meaning in every row.

cl100k_base · GPT‑4 o200k_base · GPT‑4o
0 tokens

Select any language to read the passage it was measured on. Bar length is the token count; the number at the end is that count and its multiple of English.

What it costs you

The same product, priced differently by alphabet.

Set your own usage and your provider's price. This is arithmetic on the measured token counts above, not a quote from any vendor — and it is the same request, answered the same way, for a user who happens to write in a different script.

Annual cost, English—
Annual cost, ——
Extra paid for the same sentences—
Copies of the passage that fit in context—
The part that complicates it

The newer tokenizer fixed this. For some people.

Compare the two bars in each row. Moving from cl100k_base to o200k_base — roughly, from GPT‑4 to GPT‑4o — cut Hindi's tax from 5.5× to 1.6×, Bengali's from 6.4× to 1.7×, and Telugu's from 8.5× to 2.4×. That is a real and substantial repair, and it deserves saying plainly.

Now look at Amharic: 9.2× to 6.2×. Nahuatl: 2.7× to 2.5×. Vietnamese: 4.1× to 3.4×. The languages that got the repair are the ones with large digitised corpora and commercially interesting speaker bases. The ones still paying are, broadly, the ones that were already underserved — which is the same pattern that produced the tax in the first place, one iteration later.

I argue the wider case in MyTh 002, The Default Human. The short version: nobody chose this. It fell out of an engineering decision about compression, made by people optimising for the corpus in front of them — which is how this kind of bias usually arrives.

Method, and what this is not

How these numbers were made.

  • The passage is Article 1 of the Universal Declaration of Human Rights, in professional translations from the Unicode UDHR in XML project. It is public domain and genuinely parallel — every row says the same thing — so a difference in token count is a difference in encoding, not in content.
  • The tokenizers are OpenAI's cl100k_base (GPT‑3.5, GPT‑4) and o200k_base (GPT‑4o and later), run locally through tiktoken. Other model families use different tokenizers and will give different numbers; the pattern, not the exact figure, is the point.
  • The multiplier is that language's token count divided by English's, per tokenizer.
  • The data is published. Download the JSON or read the script that generated it and re-run it yourself.

What this is not: a rigorous corpus study. It is one short paragraph of formal legal register. Translations legitimately differ in length, some languages here have several standard orthographies, and a single passage cannot capture how a tokenizer behaves across a language. Treat it as a demonstration of a finding, not the finding itself.

The finding itself is Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023), which measured this properly across a parallel corpus and reports differences of up to 15×, together with the three consequences that follow — cost, latency, and how much of your problem fits in context. arxiv.org/abs/2305.15425

More of this

If the numbers are wrong, I would rather know.

Read MyTh 002 Tell me it's wrong The research behind it