TiKMiX: Take Data Influence into Dynamic Mixture for Language Model
  Pre-training
The data mixture used in the pre-training of a language model is a
cornerstone of its final performance. However, a static mixing strategy is
suboptimal, as the model's learning preferences for various data domains shift
dynamically throughout training. Crucially, observing these evolving
preferences in a computationally efficient manner remains a significant
challenge. To address this, we propose TiKMiX, a method that dynamically
adjusts the data mixture according to the model's evolving preferences. TiKMiX
introduces Group Influence, an efficient metric for evaluating the impact of
data domains on the model. This metric enables the formulation of the data
mixing problem as a search for an optimal, influence-maximizing distribution.
We solve this via two approaches: TiKMiX-D for direct optimization, and
TiKMiX-M, which uses a regression model to predict a superior mixture. We
trained models with different numbers of parameters, on up to 1 trillion
tokens. TiKMiX-D exceeds the performance of state-of-the-art methods like
REGMIX while using just 20% of the computational resources. TiKMiX-M leads to
an average performance gain of 2% across 9 downstream benchmarks. Our
experiments reveal that a model's data preferences evolve with training
progress and scale, and we demonstrate that dynamically adjusting the data
mixture based on Group Influence, a direct measure of these preferences,
significantly improves performance by mitigating the underdigestion of data
seen with static ratios.
68aec6138a2eb0ba9b54e982
Yifan Wang
68aec6138a2eb0ba9b54e983
Binbin Liu
68aec6138a2eb0ba9b54e984
Fengze Liu
68aec6138a2eb0ba9b54e985
Yuanfan Guo
68aec6138a2eb0ba9b54e986
Jiyao Deng
68aec6138a2eb0ba9b54e987
Xuecheng Wu
68aec6138a2eb0ba9b54e988
Weidong Zhou
68aec6138a2eb0ba9b54e989
Xiaohuan Zhou
68aec6138a2eb0ba9b54e98a
Taifeng Wang
TiKMiX: Take Data Influence into Dynamic Mixture for Language Model
  Pre-training
The data mixture used in the pre-training of a language model is a
cornerstone of its final performance. However, a static mixing strategy is
suboptimal, as the model's learning preferences for various data domains shift
dynamically throughout training. Crucially, observing these evolving
preferences in a computationally efficient manner remains a significant
challenge. To address this, we propose TiKMiX, a method that dynamically
adjusts the data mixture according to the model's evolving preferences. TiKMiX
introduces Group Influence, an efficient metric for evaluating the impact of
data domains on the model. This metric enables the formulation of the data
mixing problem as a search for an optimal, influence-maximizing distribution.
We solve this via two approaches: TiKMiX-D for direct optimization, and
TiKMiX-M, which uses a regression model to predict a superior mixture. We
trained models with different numbers of parameters, on up to 1 trillion
tokens. TiKMiX-D exceeds the performance of state-of-the-art methods like
REGMIX while using just 20% of the computational resources. TiKMiX-M leads to
an average performance gain of 2% across 9 downstream benchmarks. Our
experiments reveal that a model's data preferences evolve with training
progress and scale, and we demonstrate that dynamically adjusting the data
mixture based on Group Influence, a direct measure of these preferences,
significantly improves performance by mitigating the underdigestion of data
seen with static ratios.
Dynamic adjustment of data mixture based on Group Influence metric improves language model performance by adapting to evolving learning preferences.
Group Influence TiKMiX TiKMiX-D TiKMiX-M data mixture language model pre-training computational resources downstream benchmarks data preferences underdigestion