Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining
Efficient data selection is crucial to accelerate the pretraining of large
language models (LLMs). While various methods have been proposed to enhance
data efficiency, limited research has addressed the inherent conflicts between
these approaches to achieve optimal data selection for LLM pretraining. To
tackle this problem, we propose a novel multi-agent collaborative data
selection mechanism. In this framework, each data selection method serves as an
independent agent, and an agent console is designed to dynamically integrate
the information from all agents throughout the LLM training process. We conduct
extensive empirical studies to evaluate our multi-agent framework. The
experimental results demonstrate that our approach significantly improves data
efficiency, accelerates convergence in LLM training, and achieves an average
performance gain of 10.5% across multiple language model benchmarks compared to
the state-of-the-art methods.
670cab96200342b4d2c77ff5
Tianyi Bai
claimed_verified
2024-10-14T10:47:44.353Z
641802c31f1f3b0fa812225c
https://cdn-avatars.huggingface.co/v1/production/uploads/641802c31f1f3b0fa812225c/i-OFWm8d37ofsmuvvAKYR.png
becca_bai
user
beccabai
670cab96200342b4d2c77ff6
Ling Yang
admin_assigned
2024-10-14T12:37:15.149Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
670cab96200342b4d2c77ff7
Zhen Hao Wong
admin_assigned
2024-10-14T12:38:20.132Z
65d3149f0545ab7c11568b2b
/avatars/02f36d8320b8331849e7d7f40e509ca8.svg
Zhen Hao Wong
user
aaron1141
670cab96200342b4d2c77ff8
Jiahui Peng
admin_assigned
2024-10-14T12:38:52.705Z
64aa3d4b329f5e6e780dce4a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5F_eFJ8ZZOgDxrOHSFyzJ.png
Jiahui Peng
user
Jiahui2023
670cab96200342b4d2c77ff9
Xinlin Zhuang
admin_assigned
2024-10-14T12:38:46.006Z
63567f00f44a9f0f5f885221
/avatars/bc41b72399b7a7fe9293cde62f0577a0.svg
Xinlin Zhuang
user
Mihara-bot
670cab96200342b4d2c77ffa
Chi Zhang
670cab96200342b4d2c77ffb
Lijun Wu
670cab96200342b4d2c77ffc
Qiu Jiantao
admin_assigned
2024-10-14T12:39:34.816Z
641bbd19af42e9b7dd4638ce
/avatars/8e7819fa76650cab30e60063e8d3a217.svg
Qiu Jiantao
user
qiujiantao
670cab96200342b4d2c77ffd
Wentao Zhang
670cab96200342b4d2c77ffe
Binhang Yuan
admin_assigned
2024-10-14T12:40:04.373Z
6307c8dda642348dc5e71c20
https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
Binhang Yuan
user
biyuan
670cab96200342b4d2c77fff
Conghui He
admin_assigned
2024-10-14T12:39:58.611Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining
Efficient data selection is crucial to accelerate the pretraining of large
language models (LLMs). While various methods have been proposed to enhance
data efficiency, limited research has addressed the inherent conflicts between
these approaches to achieve optimal data selection for LLM pretraining. To
tackle this problem, we propose a novel multi-agent collaborative data
selection mechanism. In this framework, each data selection method serves as an
independent agent, and an agent console is designed to dynamically integrate
the information from all agents throughout the LLM training process. We conduct
extensive empirical studies to evaluate our multi-agent framework. The
experimental results demonstrate that our approach significantly improves data
efficiency, accelerates convergence in LLM training, and achieves an average
performance gain of 10.5% across multiple language model benchmarks compared to
the state-of-the-art methods.
A multi-agent collaborative mechanism enhances data selection for pretraining large language models, improving efficiency and performance.
multi-agent collaborative data selection large language models LLM pretraining data efficiency convergence performance gain