Train Serve Consistency
A model is only as good as the match between training inputs and serving inputs. Train serve skew is any difference between them, and it silently destroys accuracy in production.
Sources of skew
- Code divergence when training uses one transformation script and serving reimplements it differently.
- Data divergence when the serving feature store is stale or computed on different windows.
- Distribution shift when production traffic drifts away from the training era.
Defenses
The strongest defense is to share the exact transformation code between training and serving, so the same function produces features in both. A feature store with a single definition enforces this. Logging the actual features served at prediction time, then training on those logged features, closes the loop and guarantees parity.
Detecting skew
Even with care, skew creeps in. Teams compare the distribution of features at training time against features observed at serving and alert on divergence. Catching skew early, before accuracy visibly drops, is what separates a robust system from one that mysteriously degrades.
Key idea
Train serve consistency means identical transformation code and matching feature distributions across training and production, with skew monitoring to catch drift.