The Epoch Batch and Iteration
Three terms describe how data flows through training, and mixing them up causes endless confusion. They are the epoch, the batch, and the iteration.
A batch is the group of examples processed together before the model updates its parameters once. Its size, the batch size, balances speed and memory against gradient quality. Bigger batches give smoother gradients but cost more memory.
An iteration is one parameter update, which corresponds to processing one batch. Each iteration runs a forward pass, computes loss, and applies an update for that batch.
An epoch is one full pass through the entire training dataset. After an epoch, the model has seen every example exactly once.
These connect with a simple formula. The number of iterations per epoch equals the dataset size divided by the batch size. So a dataset of one thousand examples with a batch size of one hundred yields ten iterations per epoch.
Why the distinction matters:
- You set epochs to control how many times the model revisits the data
- You set batch size to control memory use and gradient smoothness
- Iterations simply fall out of those two choices
Training usually spans many epochs, shuffling the data each pass so batches differ and the model does not learn the order.
Key idea
A batch is the group updated together, an iteration is one update, and an epoch is one full pass through the data; iterations per epoch equal dataset size over batch size.