1 note tagged “tokenization”
Why token count for Hindi and English differ
Hindi used to cost six times more tokens than English for the same sentence. Then it stopped. Both the problem and the fix come out of how BPE builds its vocabulary.
Why token count for Hindi and English differ
Hindi used to cost six times more tokens than English for the same sentence. Then it stopped. Both the problem and the fix come out of how BPE builds its vocabulary.