Benefits and Pitfalls of Reinforcement Learning for Language Model
  Planning: A Theoretical Perspective
Recent reinforcement learning (RL) methods have substantially enhanced the
planning capabilities of Large Language Models (LLMs), yet the theoretical
basis for their effectiveness remains elusive. In this work, we investigate
RL's benefits and limitations through a tractable graph-based abstraction,
focusing on policy gradient (PG) and Q-learning methods. Our theoretical
analyses reveal that supervised fine-tuning (SFT) may introduce
co-occurrence-based spurious solutions, whereas RL achieves correct planning
primarily through exploration, underscoring exploration's role in enabling
better generalization. However, we also show that PG suffers from diversity
collapse, where output diversity decreases during training and persists even
after perfect accuracy is attained. By contrast, Q-learning provides two key
advantages: off-policy learning and diversity preservation at convergence. We
further demonstrate that careful reward design is necessary to prevent reward
hacking in Q-learning. Finally, applying our framework to the real-world
planning benchmark Blocksworld, we confirm that these behaviors manifest in
practice.
68dc86ef4159d1f2418f9a09
Siwei Wang
68dc86ef4159d1f2418f9a0a
Yifei Shen
68dc86ef4159d1f2418f9a0b
Haoran Sun
68dc86ef4159d1f2418f9a0c
Shi Feng
68dc86ef4159d1f2418f9a0d
Shang-Hua Teng
68dc86ef4159d1f2418f9a0e
Li Dong
68dc86ef4159d1f2418f9a0f
Yaru Hao
68dc86ef4159d1f2418f9a10
Wei Chen
claimed_verified
2025-10-01T10:22:29.262Z
68dcae5c9cb88433d59a17b2
/avatars/09e90d4a22fb8ad476d0e5f440acc2be.svg
Wei Chen
user
weichen999
Benefits and Pitfalls of Reinforcement Learning for Language Model
  Planning: A Theoretical Perspective
Recent reinforcement learning (RL) methods have substantially enhanced the
planning capabilities of Large Language Models (LLMs), yet the theoretical
basis for their effectiveness remains elusive. In this work, we investigate
RL's benefits and limitations through a tractable graph-based abstraction,
focusing on policy gradient (PG) and Q-learning methods. Our theoretical
analyses reveal that supervised fine-tuning (SFT) may introduce
co-occurrence-based spurious solutions, whereas RL achieves correct planning
primarily through exploration, underscoring exploration's role in enabling
better generalization. However, we also show that PG suffers from diversity
collapse, where output diversity decreases during training and persists even
after perfect accuracy is attained. By contrast, Q-learning provides two key
advantages: off-policy learning and diversity preservation at convergence. We
further demonstrate that careful reward design is necessary to prevent reward
hacking in Q-learning. Finally, applying our framework to the real-world
planning benchmark Blocksworld, we confirm that these behaviors manifest in
practice.
Theoretical analysis of reinforcement learning methods in enhancing LLM planning reveals that while RL improves generalization through exploration, policy gradient suffers from diversity collapse, whereas Q-learning maintains diversity and requires careful reward design.
reinforcement learning Large Language Models policy gradient Q-learning supervised fine-tuning co-occurrence-based spurious solutions diversity collapse off-policy learning reward hacking Blocksworld