Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament
Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large
Language Models (LLMs), relies on reward models to select the best candidate
solution from multiple generations. However, traditional reward models often
assign arbitrary and inconsistent scores, limiting their effectiveness. To
address this, we propose a Pairwise Reward Model (Pairwise RM) combined with a
knockout tournament for BoN sampling. Instead of assigning absolute scores,
given one math problem, Pairwise RM evaluates two candidate solutions'
correctness simultaneously. This approach eliminates the need for arbitrary
scoring and enables cross-validation of solutions through parallel comparison.
In the knockout tournament, Pairwise RM conducts pairwise comparisons between
candidate solutions and eliminates the incorrect ones iteratively. We construct
\ourdataset, a large-scale dataset of 443K pairwise comparisons derived from
NumiaMath and annotated using gemini-1.5-flash, and train the Pairwise
RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench
demonstrate significant improvements over traditional discriminative reward
models. And a 40\% to 60\% relative improvement is achieved on the top 50\%
challenging problems.
6791d751f29664a338e7b4c5
Yantao Liu
6791d751f29664a338e7b4c6
Zijun Yao
6791d751f29664a338e7b4c7
Rui Min
6791d751f29664a338e7b4c8
Yixin Cao
6791d751f29664a338e7b4c9
Lei Hou
6791d751f29664a338e7b4ca
Juanzi Li
admin_assigned
2025-01-23T09:49:20.111Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament
Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large
Language Models (LLMs), relies on reward models to select the best candidate
solution from multiple generations. However, traditional reward models often
assign arbitrary and inconsistent scores, limiting their effectiveness. To
address this, we propose a Pairwise Reward Model (Pairwise RM) combined with a
knockout tournament for BoN sampling. Instead of assigning absolute scores,
given one math problem, Pairwise RM evaluates two candidate solutions'
correctness simultaneously. This approach eliminates the need for arbitrary
scoring and enables cross-validation of solutions through parallel comparison.
In the knockout tournament, Pairwise RM conducts pairwise comparisons between
candidate solutions and eliminates the incorrect ones iteratively. We construct
\ourdataset, a large-scale dataset of 443K pairwise comparisons derived from
NumiaMath and annotated using gemini-1.5-flash, and train the Pairwise
RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench
demonstrate significant improvements over traditional discriminative reward
models. And a 40\% to 60\% relative improvement is achieved on the top 50\%
challenging problems.
Pairwise Reward Model combined with a knockout tournament enhances test-time scaling of Large Language Models by accurately comparing candidate solutions through parallel evaluation.
Best-of-N (BoN) sampling Large Language Models (LLMs) reward models Pairwise Reward Model knockout tournament NumiaMath MATH-500 Olympiad Bench