ICon: In-Context Contribution for Automatic Data Selection
Data selection for instruction tuning is essential for improving the
performance of Large Language Models (LLMs) and reducing training cost.
However, existing automated selection methods either depend on computationally
expensive gradient-based measures or manually designed heuristics, which may
fail to fully exploit the intrinsic attributes of data. In this paper, we
propose In-context Learning for Contribution Measurement (ICon), a novel
gradient-free method that takes advantage of the implicit fine-tuning nature of
in-context learning (ICL) to measure sample contribution without gradient
computation or manual indicators engineering. ICon offers a computationally
efficient alternative to gradient-based methods and reduces human inductive
bias inherent in heuristic-based approaches. ICon comprises three components
and identifies high-contribution data by assessing performance shifts under
implicit learning through ICL. Extensive experiments on three LLMs across 12
benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of
ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data
outperform full datasets by 5.42% points and exceed the best performance of
widely used selection methods by 2.06% points. We further analyze
high-contribution samples selected by ICon, which show both diverse tasks and
appropriate difficulty levels, rather than just the hardest ones.
681d95bc11abe59dc97e4c5a
Yixin Yang
claimed_verified
2025-05-09T07:21:18.753Z
647e99d9becb41a272970ca4
/avatars/291831643937a298c5903c6dc037b950.svg
Ann
user
yyxsghx
681d95bc11abe59dc97e4c5b
Qingxiu Dong
admin_assigned
2025-05-09T08:57:38.844Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Rosy24
user
Rsy24
681d95bc11abe59dc97e4c5c
Linli Yao
admin_assigned
2025-05-09T08:57:47.561Z
655ca347f426a304c6b393a1
/avatars/67f0310d59c5912d38c2ad8e6448614d.svg
Linli Yao
user
yaolily
681d95bc11abe59dc97e4c5d
Fangwei Zhu
admin_assigned
2025-05-09T08:57:53.796Z
654cca3fe1b4cd6d40d5a7ae
/avatars/4d09531277e16ad71474fc888c16b227.svg
Fangwei Zhu
user
soliz1998
681d95bc11abe59dc97e4c5e
Zhifang Sui
ICon: In-Context Contribution for Automatic Data Selection
Data selection for instruction tuning is essential for improving the
performance of Large Language Models (LLMs) and reducing training cost.
However, existing automated selection methods either depend on computationally
expensive gradient-based measures or manually designed heuristics, which may
fail to fully exploit the intrinsic attributes of data. In this paper, we
propose In-context Learning for Contribution Measurement (ICon), a novel
gradient-free method that takes advantage of the implicit fine-tuning nature of
in-context learning (ICL) to measure sample contribution without gradient
computation or manual indicators engineering. ICon offers a computationally
efficient alternative to gradient-based methods and reduces human inductive
bias inherent in heuristic-based approaches. ICon comprises three components
and identifies high-contribution data by assessing performance shifts under
implicit learning through ICL. Extensive experiments on three LLMs across 12
benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of
ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data
outperform full datasets by 5.42% points and exceed the best performance of
widely used selection methods by 2.06% points. We further analyze
high-contribution samples selected by ICon, which show both diverse tasks and
appropriate difficulty levels, rather than just the hardest ones.
In-context Learning for Contribution Measurement (ICon) is a gradient-free method that identifies high-contribution data for instruction tuning of Large Language Models, outperforming gradient-based and heuristic-based selection methods.
in-context learning ICon Large Language Models LLMs gradient-free method sample contribution performance shifts implicit learning ICL