Loading
Why one word can be three tokens and why code and other languages cost more.
In a picture
A tokenizer cuts text into sub-word pieces it knows.
Each piece becomes a number the model can work with.
Tokens decide cost, speed and what fits in the window.
Stage 1 of 3: Split
Open the full conceptMade for you now: slide lessons from basics to production, usually with code and a quiz. See a finished one