Walk downhill on the loss, one step at a time; optimisers choose the step.
- LossMeasure how wrong the model is on a batch.
- GradientFind the steepest downhill direction.
- StepMove weights by learning rate x gradient.
- OptimiserAdam and momentum adapt the step per weight.
Explain 1/5
Source: PyTorch: torch.optim