Random Search Tuning
Random search tunes hyperparameters by sampling combinations at random from defined ranges instead of testing a fixed grid. With a fixed budget it often beats grid search.
How it works
- Define a distribution or range for each hyperparameter.
- Draw a fixed number of random samples from those ranges.
- Evaluate each sampled combination with cross validation and keep the best.
Why it often wins
- Most problems have a few important hyperparameters and many unimportant ones.
- Grid search wastes its budget testing many values of unimportant parameters.
- Random search spreads samples so it explores more distinct values of the important parameters for the same number of trials.
Practical tips
- Sample some parameters on a log scale, such as learning rates.
- Increase the budget until improvements flatten out.
- It parallelizes as easily as grid search, since samples are independent.
Key idea
Random search samples hyperparameter combinations from ranges, exploring more distinct values of the important parameters than grid search for the same budget.