Hold data back, pick the right metric, and catch overfitting before users do.
- SplitTrain on one part, tune on another, test once on the last.
- Cross-validateRotate which slice is held out and average the scores.
- MetricChoose precision, recall, F1 or error that matches the cost of mistakes.
- CurvesCompare training and validation scores as you train.
Explain 1/5
Source: scikit-learn: Cross-validation