Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Large language models (LLMs) have achieved remarkable performance in recent
years but are fundamentally limited by the underlying training data. To improve
models beyond the training data, recent works have explored how LLMs can be
used to generate synthetic data for autonomous self-improvement. However,
successive steps of self-improvement can reach a point of diminishing returns.
In this work, we propose a complementary approach towards self-improvement
where finetuning is applied to a multiagent society of language models. A group
of language models, all starting from the same base model, are independently
specialized by updating each one using data generated through multiagent
interactions among the models. By training each model on independent sets of
data, we illustrate how this approach enables specialization across models and
diversification over the set of models. As a result, our overall system is able
to preserve diverse reasoning chains and autonomously improve over many more
rounds of fine-tuning than single-agent self-improvement methods. We
quantitatively illustrate the efficacy of the approach across a wide suite of
reasoning tasks.
6784a4b4cf658d598c274e52
Vighnesh Subramaniam
admin_assigned
2025-01-13T09:34:56.291Z
63dc6b595d310b018be1a6c7
/avatars/f6bb94fe302b1627b27f533ba6c32ed3.svg
Vighnesh Subramaniam
user
vsub851
6784a4b4cf658d598c274e53
Yilun Du
admin_assigned
2025-01-13T09:35:03.427Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
6784a4b4cf658d598c274e54
Joshua B. Tenenbaum
6784a4b4cf658d598c274e55
Antonio Torralba
admin_assigned
2025-01-13T09:35:36.299Z
6744c6ec6ec99a37d4ba9235
/avatars/a5384b63bb615192f6fa157c6ea89e92.svg
Antonio
user
Antoniotorralbaborruel
6784a4b4cf658d598c274e56
Shuang Li
6784a4b4cf658d598c274e57
Igor Mordatch
Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Large language models (LLMs) have achieved remarkable performance in recent
years but are fundamentally limited by the underlying training data. To improve
models beyond the training data, recent works have explored how LLMs can be
used to generate synthetic data for autonomous self-improvement. However,
successive steps of self-improvement can reach a point of diminishing returns.
In this work, we propose a complementary approach towards self-improvement
where finetuning is applied to a multiagent society of language models. A group
of language models, all starting from the same base model, are independently
specialized by updating each one using data generated through multiagent
interactions among the models. By training each model on independent sets of
data, we illustrate how this approach enables specialization across models and
diversification over the set of models. As a result, our overall system is able
to preserve diverse reasoning chains and autonomously improve over many more
rounds of fine-tuning than single-agent self-improvement methods. We
quantitatively illustrate the efficacy of the approach across a wide suite of
reasoning tasks.
A multi-agent society of language models self-improves through interactions, enabling specialization and better performance over rounds of fine-tuning compared to single-agent methods.
large language models synthetic data self-improvement finetuning multiagent society independent specialization reasoning chains