Models learn language by filling blanks (MLM) or predicting the next word (CLM).
- Mask (MLM)Hide 15% of words; predict them from both sides.
- Next token (CLM)Predict each word from the words before it.
- Huge textBillions of sentences, no labels needed.
- ResultMLM gives good readers (BERT); CLM gives writers (GPT).
Explain 1/5
Source: Devlin et al.: BERT