CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom
Distilling advanced Large Language Models' instruction-following capabilities
into smaller models using a selected subset has become a mainstream approach in
model training. While existing synthetic instruction data selection strategies
rely mainly on single-dimensional signals (i.e., reward scores, model
perplexity), they fail to capture the complexity of instruction-following
across diverse fields. Therefore, we investigate more diverse signals to
capture comprehensive instruction-response pair characteristics and propose
three foundational metrics that leverage Multi-LLM wisdom, informed by (1)
diverse LLM responses and (2) reward model assessment. Building upon base
metrics, we propose CrowdSelect, an integrated metric incorporating a
clustering-based approach to maintain response diversity. Our comprehensive
experiments demonstrate that our foundation metrics consistently improve
performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect,
efficiently incorporating all metrics, achieves state-of-the-art performance in
both Full and LoRA fine-tuning, showing improvements of 4.81% on Arena-Hard and
11.1% on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring
valuable insights for future research in this direction. Code are available at
https://github.com/listentm/crowdselect.
67c94c32dd505e6a4db201a2
Yisen Li
admin_assigned
2025-03-06T10:04:09.687Z
65d9a453f20e4fc19480afba
/avatars/27bfa034a13a5e7cb5fc3b647515a201.svg
yisen li
user
yisenL
67c94c32dd505e6a4db201a3
Lingfeng Yang
67c94c32dd505e6a4db201a4
Wenxuan Shen
67c94c32dd505e6a4db201a5
Pan Zhou
67c94c32dd505e6a4db201a6
Yao Wan
67c94c32dd505e6a4db201a7
Weiwei Lin
67c94c32dd505e6a4db201a8
Dongping Chen
admin_assigned
2025-03-06T10:09:00.496Z
643be8879f5d314db2d9ed23
https://cdn-avatars.huggingface.co/v1/production/uploads/643be8879f5d314db2d9ed23/VrW2UtJ7ppOnGIYjTWd7b.png
Chen Dongping
user
shuaishuaicdp
CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom
Distilling advanced Large Language Models' instruction-following capabilities
into smaller models using a selected subset has become a mainstream approach in
model training. While existing synthetic instruction data selection strategies
rely mainly on single-dimensional signals (i.e., reward scores, model
perplexity), they fail to capture the complexity of instruction-following
across diverse fields. Therefore, we investigate more diverse signals to
capture comprehensive instruction-response pair characteristics and propose
three foundational metrics that leverage Multi-LLM wisdom, informed by (1)
diverse LLM responses and (2) reward model assessment. Building upon base
metrics, we propose CrowdSelect, an integrated metric incorporating a
clustering-based approach to maintain response diversity. Our comprehensive
experiments demonstrate that our foundation metrics consistently improve
performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect,
efficiently incorporating all metrics, achieves state-of-the-art performance in
both Full and LoRA fine-tuning, showing improvements of 4.81% on Arena-Hard and
11.1% on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring
valuable insights for future research in this direction. Code are available at
https://github.com/listentm/crowdselect.
The paper explores using a multi-dimensional approach, informed by multiple LLMs and a reward model, to select high-quality instruction data, improving smaller models' performance in instruction-following tasks.
Large Language Models instruction-following synthetic instruction data reward scores model perplexity multi-dimensional signals Multi-LLM wisdom LLM responses reward model assessment CrowdSelect clustering-based approach response diversity MT-bench Arena-Hard Full fine-tuning LoRA fine-tuning