Proportions matter
A fine tuning set is rarely one source. It blends data from many tasks, domains, and styles. The data mixture is the set of proportions assigned to each source, and it strongly shapes the resulting model behavior.
Why the mix is a knob
- Over weighting one source makes the model specialize toward it.
- Including broad data preserves general ability and reduces forgetting.
- Rare but important capabilities may need up weighting to appear at all.
The mixture is effectively a set of hyperparameters as influential as the data itself.
Setting the proportions
Balancing tensions
A good mixture balances specialization against generality. Teams tune weights using held out evaluations across all target skills, sometimes adjusting iteratively. Quality also matters: a small amount of high quality data often beats a large amount of noisy data, so filtering interacts with mixing.
Key idea
The data mixture sets the proportion of each source in fine tuning and acts as a powerful knob, balancing specialization against generality and guided by evaluation across target skills.