Notes tagged llms.https://harishchandra.net/enhttps://harishchandra.net/writing/why-hindi-costs-more-tokens-than-english/https://harishchandra.net/writing/why-hindi-costs-more-tokens-than-english/Hindi used to cost six times more tokens than English for the same sentence. Then it stopped. Both the problem and the fix come out of how BPE builds its vocabulary.Sat, 05 Sep 2026 00:00:00 GMTllmstokenizationhindi