Advancing LLM Reasoning Generalists with Preference Trees
We introduce Eurus, a suite of large language models (LLMs) optimized for
reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
state-of-the-art results among open-source models on a diverse set of
benchmarks covering mathematics, code generation, and logical reasoning
problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
comprehensive benchmarking across 12 tests covering five tasks, and achieves a
33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
benchmarks, substantially outperforming existing open-source models by margins
more than 13.3%. The strong performance of Eurus can be primarily attributed to
UltraInteract, our newly-curated large-scale, high-quality alignment dataset
specifically designed for complex reasoning tasks. UltraInteract can be used in
both supervised fine-tuning and preference learning. For each instruction, it
includes a preference tree consisting of (1) reasoning chains with diverse
planning strategies in a unified format, (2) multi-turn interaction
trajectories with the environment and the critique, and (3) pairwise data to
facilitate preference learning. UltraInteract allows us to conduct an in-depth
exploration of preference learning for reasoning tasks. Our investigation
reveals that some well-established preference learning algorithms may be less
suitable for reasoning tasks compared to their effectiveness in general
conversations. Inspired by this, we derive a novel reward modeling objective
which, together with UltraInteract, leads to a strong reward model.
660cbd7d6601973e7a2a7a53
Lifan Yuan
claimed_verified
2024-12-04T09:47:43.785Z
61506ce0b0e47650b93f79d0
/avatars/21301cd9929a1a64f3f6427413855ea8.svg
Lievan
user
lievan
660cbd7d6601973e7a2a7a54
Ganqu Cui
claimed_verified
2024-04-24T11:12:19.514Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
660cbd7d6601973e7a2a7a55
Hanbin Wang
extracted_confirmed
2025-02-04T05:04:46.617Z
6321152b8c0da827c72c7c16
https://cdn-avatars.huggingface.co/v1/production/uploads/1678783813705-6321152b8c0da827c72c7c16.jpeg
Hanbin Wang
user
hanbin
660cbd7d6601973e7a2a7a56
Ning Ding
claimed_verified
2025-01-31T09:50:56.584Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
660cbd7d6601973e7a2a7a57
Xingyao Wang
claimed_verified
2024-07-25T07:39:14.796Z
62c63f31031996c36c848504
https://cdn-avatars.huggingface.co/v1/production/uploads/62c63f31031996c36c848504/xZvH6jnozuFLXbtRSU-h1.jpeg
Xingyao Wang
user
xingyaoww
660cbd7d6601973e7a2a7a58
Jia Deng
660cbd7d6601973e7a2a7a59
Boji Shan
660cbd7d6601973e7a2a7a5a
Huimin Chen
660cbd7d6601973e7a2a7a5b
Ruobing Xie
claimed_verified
2024-11-05T09:02:06.930Z
6622443b9b0614a760dd8123
/avatars/acb6c1c9c429af1112530dcf76a8e420.svg
Ruobing Xie
user
Ruobing-Xie
660cbd7d6601973e7a2a7a5c
Yankai Lin
660cbd7d6601973e7a2a7a5d
Zhenghao Liu
660cbd7d6601973e7a2a7a5e
Bowen Zhou
660cbd7d6601973e7a2a7a5f
Hao Peng
660cbd7d6601973e7a2a7a60
Zhiyuan Liu
claimed_verified
2025-06-02T07:59:36.961Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
660cbd7d6601973e7a2a7a61
Maosong Sun
Advancing LLM Reasoning Generalists with Preference Trees
We introduce Eurus, a suite of large language models (LLMs) optimized for
reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
state-of-the-art results among open-source models on a diverse set of
benchmarks covering mathematics, code generation, and logical reasoning
problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
comprehensive benchmarking across 12 tests covering five tasks, and achieves a
33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
benchmarks, substantially outperforming existing open-source models by margins
more than 13.3%. The strong performance of Eurus can be primarily attributed to
UltraInteract, our newly-curated large-scale, high-quality alignment dataset
specifically designed for complex reasoning tasks. UltraInteract can be used in
both supervised fine-tuning and preference learning. For each instruction, it
includes a preference tree consisting of (1) reasoning chains with diverse
planning strategies in a unified format, (2) multi-turn interaction
trajectories with the environment and the critique, and (3) pairwise data to
facilitate preference learning. UltraInteract allows us to conduct an in-depth
exploration of preference learning for reasoning tasks. Our investigation
reveals that some well-established preference learning algorithms may be less
suitable for reasoning tasks compared to their effectiveness in general
conversations. Inspired by this, we derive a novel reward modeling objective
which, together with UltraInteract, leads to a strong reward model.
Eurus, a suite of reasoning-optimized large language models, achieves state-of-the-art performance on various benchmarks through UltraInteract, a large-scale, high-quality alignment dataset, and a novel reward modeling objective.
large language models (LLMs) Mistral-7B CodeLlama-70B reasoning fine-tuning UltraInteract benchmarking LeetCode TheoremQA preference learning preference tree reasoning chains reward modeling