The tokenization pipeline

What happens when we call a tokenizer on some texts, and how does it compute the numbers it outputs? This video is part of the …