The per-token feed-forward block that stores most of the model’s knowledge.
Made for you now: slide lessons from basics to production, usually with code and a quiz. See a finished one
After attention mixes information between tokens, the feed-forward network (FFN) processes each token on its own with the same two-layer network: expand to a wider size, apply a non-linearity, project back.
Attention decides what to look at; the FFN transforms what was gathered. In most transformers the FFN holds the majority of the block’s parameters.
Widening the hidden layer (commonly several times the model width) gives the network room to represent many features before compressing again. Interpretability work suggests these layers act like key-value memories for learned patterns, which is a hypothesis, not settled fact.
Gated variants such as SwiGLU replace the plain activation with a gate that multiplies two projections. Mixture-of-experts models replace one FFN with many and route each token to a few, adding capacity without running every parameter per token.
Step 1
Token states in
One vector per token from the previous layer.
Step 2
Self-attention
Tokens exchange information.
Step 3
Add + normalise
Residual connection keeps the original signal.
Step 4
FFN
Expand, non-linearity, project back, per token.
Step 5
Add + normalise
Output passes to the next block.