Why token count for Hindi and English differ

To understand this, we have to first understand what a token is.

Nvidia defines it as “Tiny units of data that come from breaking down bigger chunks of information.”

“Breaking down” is the interesting part, because it decides how much we have to break the data.

There are a few ways this can be done. Say we have to break this data:

send the file to the team by three pm

We can break it at the character level. That would give us:

['s', 'e', 'n', 'd', ' ', 't', 'h', 'e', ' ', 'f', 'i', 'l', 'e', ' ', 't', 'o',
 ' ', 't', 'h', 'e', ' ', 't', 'e', 'a', 'm', ' ', 'b', 'y', ' ', 't', 'h', 'r',
 'e', 'e', ' ', 'p', 'm']

The other way would be word level:

['send', 'the', 'file', 'to', 'the', 'team', 'by', 'three', 'pm']

Both have a problem:

• Character level: long sequence, and it loses the contextual meaning.
• Word level: struggles with rare and unknown words.

BPE

BPE is the solution, and the one which gets used by the GPTs of the world. Here GPT means Generative Pre-trained Transformer, not OpenAI’s model, but that counts too, as the brand name came from the same place.

BPE is Byte-Pair-Encoding. Its job is to find commonly occurring subwords. It is better than both approaches because it is able to reduce the long sequence of words but still able to keep the context and unknown words.

For it to work, it needs to be trained on a large corpus of data, so that it can distil that and create its vocab.

I first pictured this as a hierarchy that goes downward: check the vocab for the word, and if it is not there, cut it into subwords, and if those are not there either, cut it into characters.

It is the other way around. BPE starts at the smallest pieces and merges upward. During training it repeatedly finds the most frequent adjacent pair and merges it, thousands of times over. Tokenizing is just replaying those merges in the order they were learned.

The result looks like the hierarchy I imagined, which is why it is an easy picture to land on:

1. A common word is one token, because every merge along the way fired.
2. A rarer word is a few subwords, because the merges stopped partway.
3. A word the tokenizer never saw stays in pieces, because no merge fired at all.

The difference is that nothing is being cut. Things are failing to be joined.

Read bottom to top. The merges either fire or they do not.

What that does to Hindi

Most of Hindi text was rare for these models. Few frequent adjacent pairs means few merges won, which means Hindi never got promoted into whole-word tokens.

And the floor it falls back to is worse than characters. It is called Byte Pair Encoding because it works on UTF-8 bytes, and Devanagari is three bytes per character where English is one. नमस्ते is six characters but eighteen bytes.

Here is GPT-2 tokenizing a Hindi sentence:

['�', '�', '�', '�', 'ा', '�', '�', '�', '�', ' �', '�', '�', ...]

Those broken symbols are not characters. They are pieces of characters.

The numbers

Same sentence in both languages, send the file to the team by three pm against फ़ाइल टीम को तीन बजे भेज दो:

Tokenizer	Model	English	Hindi
r50k_base	GPT-2, GPT-3	9	41
cl100k_base	GPT-3.5, GPT-4	9	30
o200k_base	GPT-4o and newer	9	8

On the current tokenizer the Hindi sentence splits like this:

['फ़', 'ाइल', ' टीम', ' को', ' तीन', ' बजे', ' भेज', ' दो']

टीम, को, तीन, बजे, भेज, दो. Whole words, one token each.

The fix was the obvious one

Train the tokenizer on a corpus with enough Hindi in it so those merges get a chance to win. That is what happened between cl100k and o200k.

The vocabulary is fixed when the tokenizer is trained, so a new vocabulary means a new tokenizer, and the model has to be built around it.

In short

Frequency determines merge order, merge order determines who gets to be a single token.

Devanagari was rare in the training text → few frequent adjacent pairs → few merges won → Hindi stayed near single bytes. English was everywhere → merges ran all the way up → whole common words are one token.

The interesting part is that this was never a property of the language. It was a property of the training data, which is why it could be fixed by changing the training data.

Reproducing this

import tiktoken

en = "send the file to the team by three pm"
hi = "फ़ाइल टीम को तीन बजे भेज दो"

for name in ["r50k_base", "cl100k_base", "o200k_base"]:
    enc = tiktoken.get_encoding(name)
    print(name, len(enc.encode(en)), len(enc.encode(hi)))

Links

• Tiktokenizer
• BPE from scratch