Labels are the bottleneck
Retraining needs fresh labeled data, but labeling is slow and costly. The skill is choosing which production samples to label so each annotation buys the most improvement, not labeling everything blindly.
Smart sampling strategies
- Uncertainty sampling, label cases where the model is least confident, near the decision boundary.
- Drift focused, label recent data from regions where inputs have shifted.
- Stratified, ensure rare classes and key segments get coverage.
- Disagreement, label where a new candidate model disagrees with production.
Keeping labels trustworthy
- Measure inter annotator agreement to catch ambiguous guidelines.
- Use clear instructions and adjudication for hard cases.
- Audit a sample of labels, since bad labels teach the model wrong answers.
Feeding retraining
Combine new labels with existing data, watching the class balance and freshness. The goal is a training set that reflects today's world, not last year's.
Key idea
Labeling for retraining is an active learning problem, selecting uncertain, drifted, or disagreement samples and auditing label quality so each annotation maximally improves the next model.