Efficient Data Selection at Scale via Influence Distillation
Effective data selection is critical for efficient training of modern Large
Language Models (LLMs). This paper introduces Influence Distillation, a novel,
mathematically-justified framework for data selection that employs second-order
information to optimally weight training samples. By distilling each sample's
influence on a target distribution, our method assigns model-specific weights
that are used to select training data for LLM fine-tuning, guiding it toward
strong performance on the target domain. We derive these optimal weights for
both Gradient Descent and Adam optimizers. To ensure scalability and reduce
computational cost, we propose a landmark-based approximation:
influence is precisely computed for a small subset of "landmark" samples and
then efficiently propagated to all other samples to determine their weights. We
validate Influence Distillation by applying it to instruction tuning on the
Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU,
across several models from the Llama and Qwen families. Experiments show that
Influence Distillation matches or outperforms state-of-the-art performance
while achieving up to 3.5times faster selection.
68383ce6023315653bf9c3be
Mahdi Nikdan
claimed_verified
2025-05-29T20:19:11.613Z
6526b8ebba9a8279c139616b
/avatars/09f6b677603a03be128996a0765233e6.svg
Mahdi Nikdan
user
mnikdan97
68383ce6023315653bf9c3bf
Vincent Cohen-Addad
68383ce6023315653bf9c3c0
Dan Alistarh
68383ce6023315653bf9c3c1
Vahab Mirrokni
Efficient Data Selection at Scale via Influence Distillation
Effective data selection is critical for efficient training of modern Large
Language Models (LLMs). This paper introduces Influence Distillation, a novel,
mathematically-justified framework for data selection that employs second-order
information to optimally weight training samples. By distilling each sample's
influence on a target distribution, our method assigns model-specific weights
that are used to select training data for LLM fine-tuning, guiding it toward
strong performance on the target domain. We derive these optimal weights for
both Gradient Descent and Adam optimizers. To ensure scalability and reduce
computational cost, we propose a landmark-based approximation:
influence is precisely computed for a small subset of "landmark" samples and
then efficiently propagated to all other samples to determine their weights. We
validate Influence Distillation by applying it to instruction tuning on the
Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU,
across several models from the Llama and Qwen families. Experiments show that
Influence Distillation matches or outperforms state-of-the-art performance
while achieving up to 3.5times faster selection.
Influence Distillation, using second-order information, optimally selects training data for LLM fine-tuning with landmark-based approximation, achieving faster and competitive performance on various tasks.
Influence Distillation second-order information model-specific weights Gradient Descent Adam optimizers landmark-based approximation LLM fine-tuning instruction tuning Tulu V2 dataset GSM8k SQuAD MMLU Llama Qwen