Loading
Stopping a token from seeing the future.
Made for you now: slide lessons from basics to production, usually with code and a quiz. See a finished one
A text generator predicts token t from tokens before it. During training the whole sequence is visible at once, so without a mask a token could read its own answer.
Scores for future positions are set to negative infinity before softmax, so their weights become exactly zero. The result is a lower-triangular weight matrix.
Step 1
Score matrix
All token pairs.
Step 2
Set future to minus infinity
Upper triangle blocked.
Step 3
Softmax
Blocked weights become 0.
Step 4
Weighted sum of V
Each token sees only the past.