Generative Versus Discriminative Models
Machine learning models split into two camps based on what probability they learn. This choice shapes what the model can do beyond plain classification.
Two ways to model data
- A discriminative model learns the conditional probability of a label given an input. It draws decision boundaries and answers questions like is this email spam.
- A generative model learns the joint probability of inputs and labels, or just the distribution of inputs. It can sample brand new examples that resemble the training data.
Why the difference matters
- Discriminative models like logistic regression often classify more accurately with limited data because they solve the easier problem directly.
- Generative models like a variational autoencoder can create images, fill in missing values, and detect anomalies because they understand how data is shaped.
- You can turn a generative model into a classifier using Bayes rule, but not the reverse.
The recent wave of image and text synthesis comes almost entirely from generative models that learn the full distribution.
Key idea
Discriminative models learn boundaries between classes, while generative models learn the data distribution itself and can sample new examples from it.