Teach a model which answers people prefer, using pairs of better and worse replies.
- Two answersGenerate two replies to the same prompt.
- PreferenceA person picks the better one.
- Reward or directRLHF trains a reward model; DPO uses the pairs directly.
- UpdateShift the model towards preferred answers.
Explain 1/5
Source: Hugging Face TRL: DPO Trainer