Why monitor
A deployed model is not done. Inputs drift, upstream pipelines break, and accuracy decays silently. Monitoring is the practice of continuously watching a live model so problems are caught quickly.
What to monitor
Monitoring spans several layers.
- Operational health such as latency, throughput, and error rate
- Input data quality such as missing values, schema changes, or drift
- Prediction distribution such as a sudden shift in the share of positive predictions
- Model quality such as accuracy, once true labels arrive
The label delay problem
Quality metrics need ground truth, which often arrives late. You may learn the true outcome of a loan only months later. So monitoring leans heavily on proxy signals like input drift and prediction shifts that are available immediately, while true accuracy is tracked when labels land.
Alerting
You set thresholds and alert when a metric crosses them, such as drift above a limit or error rate spiking. A good alert points to a likely cause and a clear action, such as retrain or roll back, rather than just firing noise.
Key idea
Monitoring watches operations, data, and predictions so decay is caught through proxy signals long before true accuracy confirms it.