Offline Reinforcement Learning for LLM Multi-Step Reasoning
Improving the multi-step reasoning ability of large language models (LLMs)
with offline reinforcement learning (RL) is essential for quickly adapting them
to complex tasks. While Direct Preference Optimization (DPO) has shown promise
in aligning LLMs with human preferences, it is less suitable for multi-step
reasoning tasks because (1) DPO relies on paired preference data, which is not
readily available for multi-step reasoning tasks, and (2) it treats all tokens
uniformly, making it ineffective for credit assignment in multi-step reasoning
tasks, which often come with sparse reward. In this work, we propose OREO
(Offline Reasoning Optimization), an offline RL method for enhancing LLM
multi-step reasoning. Building on insights from previous works of maximum
entropy reinforcement learning, it jointly learns a policy model and value
function by optimizing the soft Bellman Equation. We show in principle that it
reduces the need to collect pairwise data and enables better credit assignment.
Empirically, OREO surpasses existing offline learning methods on multi-step
reasoning benchmarks, including mathematical reasoning tasks (GSM8K, MATH) and
embodied agent control (ALFWorld). The approach can be extended to a
multi-iteration framework when additional resources are available. Furthermore,
the learned value function can be leveraged to guide the tree search for free,
which can further boost performance during test time.
6768f050bf7c0f8d9a17c4f2
Huaijie Wang
admin_assigned
2024-12-23T11:31:56.207Z
6310664063b70252b4779150
/avatars/d514270c57b6ac494e0a419d792a72e5.svg
Huaijie Wang
user
jwhj
6768f050bf7c0f8d9a17c4f3
Shibo Hao
admin_assigned
2024-12-23T11:32:02.176Z
660ee5df35d092e3fc2a3685
/avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
Shibo Hao
user
Shibo-UCSD
6768f050bf7c0f8d9a17c4f4
Hanze Dong
admin_assigned
2024-12-23T11:32:07.779Z
63a3ff69f91ad3ea5703841d
/avatars/69227c4bce01d33747c1377b6f9672db.svg
Hanze Dong
user
hendrydong
6768f050bf7c0f8d9a17c4f5
Shenao Zhang
claimed_verified
2024-12-23T11:09:50.829Z
661213f894e0b3bff3e80c69
/avatars/d8febbb081825bf91e487aa8bad3a391.svg
Shenao Zhang
user
ZhangShenao
6768f050bf7c0f8d9a17c4f6
Yilin Bao
6768f050bf7c0f8d9a17c4f7
Ziran Yang
claimed_verified
2025-07-15T19:14:18.222Z
62d8bb07c60d1450a1eceb6f
/avatars/86df0ac603100c4b40fbde0998a5cf6f.svg
zrrr
user
zrrr
6768f050bf7c0f8d9a17c4f8
Yi Wu
admin_assigned
2024-12-23T11:32:27.882Z
62c88b04ab9c23f5c459ed90
https://cdn-avatars.huggingface.co/v1/production/uploads/62c88b04ab9c23f5c459ed90/tEaeuKpXdXwqK-zq1H-8a.png
Yi Wu
user
yiwu
Offline Reinforcement Learning for LLM Multi-Step Reasoning
Improving the multi-step reasoning ability of large language models (LLMs)
with offline reinforcement learning (RL) is essential for quickly adapting them
to complex tasks. While Direct Preference Optimization (DPO) has shown promise
in aligning LLMs with human preferences, it is less suitable for multi-step
reasoning tasks because (1) DPO relies on paired preference data, which is not
readily available for multi-step reasoning tasks, and (2) it treats all tokens
uniformly, making it ineffective for credit assignment in multi-step reasoning
tasks, which often come with sparse reward. In this work, we propose OREO
(Offline Reasoning Optimization), an offline RL method for enhancing LLM
multi-step reasoning. Building on insights from previous works of maximum
entropy reinforcement learning, it jointly learns a policy model and value
function by optimizing the soft Bellman Equation. We show in principle that it
reduces the need to collect pairwise data and enables better credit assignment.
Empirically, OREO surpasses existing offline learning methods on multi-step
reasoning benchmarks, including mathematical reasoning tasks (GSM8K, MATH) and
embodied agent control (ALFWorld). The approach can be extended to a
multi-iteration framework when additional resources are available. Furthermore,
the learned value function can be leveraged to guide the tree search for free,
which can further boost performance during test time.
OREO, an offline reinforcement learning method, enhances multi-step reasoning in large language models by optimizing a policy model and value function, improving performance on benchmarks without the need for paired preference data.
Direct Preference Optimization (DPO) offline reinforcement learning (RL) multi-step reasoning maximum entropy reinforcement learning soft Bellman Equation policy model value function GSM8K MATH ALFWorld tree search