Stronger Models are NOT Stronger Teachers for Instruction Tuning
Instruction tuning has been widely adopted to ensure large language models
(LLMs) follow user instructions effectively. The resulting
instruction-following capabilities of LLMs heavily rely on the instruction
datasets used for tuning. Recently, synthetic instruction datasets have emerged
as an economically viable solution to provide LLMs diverse and high-quality
instructions. However, existing approaches typically assume that larger or
stronger models are stronger teachers for instruction tuning, and hence simply
adopt these models as response generators to the synthetic instructions. In
this paper, we challenge this commonly-adopted assumption. Our extensive
experiments across five base models and twenty response generators reveal that
larger and stronger models are not necessarily stronger teachers of smaller
models. We refer to this phenomenon as the Larger Models' Paradox. We observe
that existing metrics cannot precisely predict the effectiveness of response
generators since they ignore the compatibility between teachers and base models
being fine-tuned. We thus develop a novel metric, named as
Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response
generators. Our experiments across five base models demonstrate that CAR
outperforms almost all baselines.
6734679ad3521b36246eb784
Zhangchen Xu
claimed_verified
2024-11-15T09:24:56.226Z
653df1323479e9ebbe3eb6cc
https://cdn-avatars.huggingface.co/v1/production/uploads/653df1323479e9ebbe3eb6cc/K_g-r1iMRNKj99LXPuYF3.jpeg
Zhangchen Xu
user
zhangchenxu
6734679ad3521b36246eb785
Fengqing Jiang
claimed_verified
2024-11-15T09:25:03.632Z
6531e1021dd8ebbdc1a6fd8e
https://cdn-avatars.huggingface.co/v1/production/uploads/6531e1021dd8ebbdc1a6fd8e/lIcl7zCPtzRsfiUh6uY1o.jpeg
Fengqing Jiang
user
fqjiang
6734679ad3521b36246eb786
Luyao Niu
claimed_verified
2025-07-15T19:14:30.100Z
666dfd4770f5a2cb4aefd12f
/avatars/fa0e0dbc203a21e58dda8fdb4cbc67ad.svg
Luyao Niu
user
LNIU
6734679ad3521b36246eb787
Bill Yuchen Lin
extracted_pending
2024-11-13T08:47:23.435Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
6734679ad3521b36246eb788
Radha Poovendran
Stronger Models are NOT Stronger Teachers for Instruction Tuning
Instruction tuning has been widely adopted to ensure large language models
(LLMs) follow user instructions effectively. The resulting
instruction-following capabilities of LLMs heavily rely on the instruction
datasets used for tuning. Recently, synthetic instruction datasets have emerged
as an economically viable solution to provide LLMs diverse and high-quality
instructions. However, existing approaches typically assume that larger or
stronger models are stronger teachers for instruction tuning, and hence simply
adopt these models as response generators to the synthetic instructions. In
this paper, we challenge this commonly-adopted assumption. Our extensive
experiments across five base models and twenty response generators reveal that
larger and stronger models are not necessarily stronger teachers of smaller
models. We refer to this phenomenon as the Larger Models' Paradox. We observe
that existing metrics cannot precisely predict the effectiveness of response
generators since they ignore the compatibility between teachers and base models
being fine-tuned. We thus develop a novel metric, named as
Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response
generators. Our experiments across five base models demonstrate that CAR
outperforms almost all baselines.
The Larger Models' Paradox reveals that larger models are not always better teachers for fine-tuning smaller models, and a new metric, Compatibility-Adjusted Reward (CAR), is introduced to measure and improve the effectiveness of response generators.
instruction tuning large language models (LLMs) instruction-following capabilities instruction datasets synthetic instruction datasets response generators Larger Models' Paradox Compatibility-Adjusted Reward (CAR)