LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
  Mathematical Reasoning
This paper presents an advanced mathematical problem-solving framework,
LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language
Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with
iterative Self-Refine to optimize the reasoning path and utilizes a pairwise
reward model to evaluate different paths globally. By leveraging the
self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS
(SR-MCTS) overcomes the inefficiencies and limitations of conventional
step-wise and greedy search algorithms by fostering a more efficient
exploration of solution spaces. Pairwise Preference Reward Model~(PPRM),
inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to
model pairwise preferences between solutions, utilizing an Enhanced Borda Count
(EBC) method to synthesize these preferences into a global ranking score to
find better answers. This approach addresses the challenges of scoring
variability and non-independent distributions in mathematical reasoning tasks.
The framework has been tested on general and advanced benchmarks, showing
superior performance in terms of search efficiency and problem-solving
capability compared to existing methods like ToT and rStar, particularly in
complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.
6704ec6a0fbef97683f15843
Di Zhang
admin_assigned
2024-10-08T09:43:53.492Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
6704ec6a0fbef97683f15844
Jianbo Wu
admin_assigned
2024-10-08T09:43:59.714Z
6407d3d3a7bc7c3865addc4b
https://cdn-avatars.huggingface.co/v1/production/uploads/6407d3d3a7bc7c3865addc4b/NEMP6xShRq-5H_rQ5ganY.jpeg
Jianbo Wu
user
jwu323
6704ec6a0fbef97683f15845
Jingdi Lei
6704ec6a0fbef97683f15846
Tong Che
6704ec6a0fbef97683f15847
Jiatong Li
claimed_verified
2024-10-09T11:21:05.725Z
6438e55cb2ea24b52ebc45ec
/avatars/e13c7398f77e7e0bd5eed03102aa5c36.svg
Jiatong LI
user
phenixace
6704ec6a0fbef97683f15848
Tong Xie
6704ec6a0fbef97683f15849
Xiaoshui Huang
6704ec6a0fbef97683f1584a
Shufei Zhang
6704ec6a0fbef97683f1584b
Marco Pavone
6704ec6a0fbef97683f1584c
Yuqiang Li
6704ec6a0fbef97683f1584d
Wanli Ouyang
6704ec6a0fbef97683f1584e
Dongzhan Zhou
admin_assigned
2024-10-08T09:47:27.102Z
6538b861613fe158bd581e35
/avatars/6817dbfe903675721fd227058b0a91ac.svg
Dongzhan Zhou
user
schrodingers-tiger
LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
  Mathematical Reasoning
This paper presents an advanced mathematical problem-solving framework,
LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language
Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with
iterative Self-Refine to optimize the reasoning path and utilizes a pairwise
reward model to evaluate different paths globally. By leveraging the
self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS
(SR-MCTS) overcomes the inefficiencies and limitations of conventional
step-wise and greedy search algorithms by fostering a more efficient
exploration of solution spaces. Pairwise Preference Reward Model~(PPRM),
inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to
model pairwise preferences between solutions, utilizing an Enhanced Borda Count
(EBC) method to synthesize these preferences into a global ranking score to
find better answers. This approach addresses the challenges of scoring
variability and non-independent distributions in mathematical reasoning tasks.
The framework has been tested on general and advanced benchmarks, showing
superior performance in terms of search efficiency and problem-solving
capability compared to existing methods like ToT and rStar, particularly in
complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.
LLaMA-Berry enhances LLMs for mathematical reasoning by combining Monte Carlo Tree Search with Self-Refine and leveraging a Pairwise Preference Reward Model to optimize search efficiency and problem-solving capability.
Monte Carlo Tree Search Self-Refine SR-MCTS Pairwise Preference Reward Model RLHF Enhanced Borda Count GPQA AIME24 AMC23 ToT rStar