minbpe vs turboBPE: Two ways to think about tokenizer training

«If you have spent time understanding how LLMs process text, you have probably come across Byte Pair Encoding. It is the algorithm sitting quietly under the hood of GPT, Llama, Mistral, and most other major models, turnin...»