The data mixture used in the pre-training of a language model is a
cornerstone of its final performance. However, a static mixing strategy is
suboptimal, as the model’s learning preferences for various data domains shift
dynamically throughout training. Crucially, observing these evolving
preferences in a computationally efficient manner remains a significant
challenge. To address this, we propose TiKMiX, a method that dynamically
adjusts the data mixture according to the model’s evolving preferences. TiKMiX
introduces Group Influence, an efficient metric for evaluating the impact of
data domains on the model. This metric enables the formulation of the data
mixing problem as a search for an optimal, influence-maximizing distribution.
We solve this via two approaches: TiKMiX-D for direct optimization, and
TiKMiX-M, which uses a regression model to predict a superior mixture. We
trained models with different numbers of parameters, on up to 1 trillion
tokens. TiKMiX-D exceeds the performance of state-of-the-art methods like
REGMIX while using just 20% of the computational resources. TiKMiX-M leads to
an average performance gain of 2% across 9 downstream benchmarks. Our
experiments reveal that a model’s data preferences evolve with training
progress and scale, and we demonstrate that dynamically adjusting the data
mixture based on Group Influence, a direct measure of these preferences,
significantly improves performance by mitigating the underdigestion of data
seen with static ratios.