Thread · HN Continuity

I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.

Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:

  Language              Relative tokens
  --------------------------------------
  English                    1.00x
  Portuguese                 1.23x
  Chinese (Simplified)       1.25x
  German                     1.31x
  Spanish                    1.32x
  French                     1.37x
  Arabic                     1.38x
  Chinese (Traditional)      1.42x
  Korean                     1.47x
  Swahili                    1.49x
  Hindi                      1.57x
  Japanese                   1.66x
  Burmese                    3.16x
  Amharic                    5.78x
  Santali                   13.70x

Source: "Tokenizer Fairness in 2026", a reproduction/extension of Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.

<a href="https://github.com/partyfly/tokenizer-fairness-2026" rel="nofollow">https://github.com/partyfly/tokenizer-fairness-2026