The need
A single train and test split gives a noisy estimate of performance that depends heavily on which rows landed in the test set. Cross validation averages over several splits for a more reliable estimate.
K fold cross validation
In k fold cross validation you split the data into k equal parts called folds. You train k times, each time holding out one fold for validation and training on the rest. Averaging the k scores gives a stable estimate of how the model generalizes.
Variants
- Stratified k fold preserves class proportions in each fold, important for imbalanced data
- Leave one out uses a single example as the validation set, very thorough but expensive
- For time series, folds must respect time order to avoid using the future to predict the past
Tuning with it
Cross validation is the standard way to choose hyperparameters like regularization strength, because it estimates generalization without touching the final test set.
Key idea
Cross validation rotates the held out fold across the data to produce a stable estimate of generalization and to tune hyperparameters.