Loading
How scores become weights and how sharpness is controlled.
Made for you now: slide lessons from basics to production, usually with code and a quiz. See a finished one
Softmax exponentiates each score and divides by the total, so outputs are positive and sum to 1. Bigger scores get disproportionately more weight.
Divide scores by a temperature T before softmax. T below 1 sharpens the distribution toward the top item; T above 1 flattens it. The same knob controls randomness when sampling the next token.
Step 1
Raw scores
Any real numbers.
Step 2
Divide by T
Low T sharpens, high T flattens.
Step 3
Softmax
Positive, sums to 1.