The Majority is not always right: RL training for solution aggregation
Scaling up test-time compute, by generating multiple independent solutions
and selecting or aggregating among them, has become a central paradigm for
improving large language models (LLMs) on challenging reasoning tasks. While
most prior work relies on simple majority voting or reward model ranking to
aggregate solutions, these approaches may only yield limited benefits. In this
work, we propose to learn aggregation as an explicit reasoning skill: given a
set of candidate solutions, we train an aggregator model to review, reconcile,
and synthesize a final, correct answer using reinforcement learning from
verifiable rewards. A key ingredient is careful balancing of easy and hard
training examples, allowing the model to learn both to recover
minority-but-correct answers as well as easy majority-correct answers.
Empirically, we find our method, AggLM, outperforms both strong rule-based and
reward-model baselines, across multiple benchmarks. Furthermore, it generalizes
effectively to solutions from differing models, including stronger ones than
contained in the training data, all while requiring substantially fewer tokens
than majority voting with larger numbers of solutions.
68bfa818207285de11b07c76
Wenting Zhao
68bfa818207285de11b07c77
Pranjal Aggarwal
68bfa818207285de11b07c78
Swarnadeep Saha
68bfa818207285de11b07c79
Asli Celikyilmaz
68bfa818207285de11b07c7a
Jason Weston
68bfa818207285de11b07c7b
Ilia Kulikov
The Majority is not always right: RL training for solution aggregation
Scaling up test-time compute, by generating multiple independent solutions
and selecting or aggregating among them, has become a central paradigm for
improving large language models (LLMs) on challenging reasoning tasks. While
most prior work relies on simple majority voting or reward model ranking to
aggregate solutions, these approaches may only yield limited benefits. In this
work, we propose to learn aggregation as an explicit reasoning skill: given a
set of candidate solutions, we train an aggregator model to review, reconcile,
and synthesize a final, correct answer using reinforcement learning from
verifiable rewards. A key ingredient is careful balancing of easy and hard
training examples, allowing the model to learn both to recover
minority-but-correct answers as well as easy majority-correct answers.
Empirically, we find our method, AggLM, outperforms both strong rule-based and
reward-model baselines, across multiple benchmarks. Furthermore, it generalizes
effectively to solutions from differing models, including stronger ones than
contained in the training data, all while requiring substantially fewer tokens
than majority voting with larger numbers of solutions.
A reinforcement learning approach to aggregating multiple solutions for large language models improves performance on reasoning tasks by learning to synthesize correct answers from candidate solutions.
large language models reinforcement learning aggregation reasoning tasks verifiable rewards AggLM