The Training Loop
Almost all modern model training follows the same rhythm, often called the training loop. Understanding it once unlocks how nearly every algorithm learns.
Each step performs four actions:
- Forward pass feeds inputs through the model to produce predictions
- Loss computation measures how wrong those predictions are against the labels
- Backward pass computes how each parameter contributed to the error
- Update nudges the parameters to reduce the loss
This cycle repeats many times. With each iteration the model's predictions inch closer to the targets, and the loss generally falls. The process stops when the loss plateaus, a time budget runs out, or validation performance stops improving.
The loop relies on the gradient, a vector pointing in the direction that increases the loss fastest. The update step moves parameters in the opposite direction, scaled by the learning rate.
A key property is that the loop is iterative and local. It never solves the whole problem at once. It makes a small improvement, re measures, and repeats. That patience is what lets it handle models with millions of parameters.
Key idea
Training repeats a forward pass, loss measurement, gradient computation, and parameter update until the model stops improving.