Reusing past keys and values so decoding is not recomputed.
Made for you now: slide lessons from basics to production, usually with code and a quiz. See a finished one
When generating token by token, keys and values of earlier tokens never change. Recomputing them at every step repeats the same work.
Store each layer’s K and V for processed tokens. For a new token compute only its own Q, K and V, append K and V to the cache, and attend over the whole cache.
The cache grows with context length, batch size and layers, so memory, not compute, often limits how many users a server can hold at once.
Step 1
New token
Compute its Q, K, V.
Step 2
Append K, V to cache
Past entries untouched.
Step 3
Attend over cache
Q against all cached K, mix cached V.
Step 4
Next token
Repeat with a bigger cache.