The architecture behind every modern LLM.
Starting with Neural network basics, Token vectors, Attention, FFNN, The transformer block, Generating one token at a time, and more
Made for you now: slide lessons from basics to production, usually with code and a quiz. See a finished one
The architecture behind every modern LLM.
AI EngineeringStage 3 of 17: Transformers
A transformer turns text into tokens, tokens into vectors, then repeats one block many times. Each block lets tokens exchange information (attention) and then transforms each token on its own (feed-forward).
Attention looks at every token pair in one matrix operation, so training parallelises well on accelerators, unlike older recurrent models that read one token at a time.
Learn vectors first, then attention, then the feed-forward block. The remaining pieces here (masking, caches, heads) refine attention.
Step 1
Tokens to vectors
Embed each token and add position.
Step 2
Attention
Tokens gather context from each other.
Step 3
Feed-forward
Each token is transformed on its own.
Step 4
Next-token scores
The last layer scores the vocabulary.
↺ Stack the block many times before scoring
Start with these 6. 10 more are one tap away.
Layers, weights and activations: the parts every model is built from.
How the model represents each token inside the network, before attention mixes them.
A model reading "the bank by the river" must decide which other words matter for each word. Attention is the mechanism that lets every token look at every other token and pull in the information it needs.
The per-token feed-forward block that stores most of the model’s knowledge.
Attention, feed-forward and norms in one unit, stacked dozens of times.
Each new token is fed back in as input for the next one.