Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
Parallel thinking has emerged as a novel approach for enhancing the reasoning
capabilities of large language models (LLMs) by exploring multiple reasoning
paths concurrently. However, activating such capabilities through training
remains challenging, as existing methods predominantly rely on supervised
fine-tuning (SFT) over synthetic data, which encourages teacher-forced
imitation rather than exploration and generalization. Different from them, we
propose Parallel-R1, the first reinforcement learning (RL) framework
that enables parallel thinking behaviors for complex real-world reasoning
tasks. Our framework employs a progressive curriculum that explicitly addresses
the cold-start problem in training parallel thinking with RL. We first use SFT
on prompt-generated trajectories from easier tasks to instill the parallel
thinking ability, then transition to RL to explore and generalize this skill on
harder problems. Experiments on various math benchmarks, including MATH, AMC23,
and AIME, show that Parallel-R1 successfully instills parallel thinking,
leading to 8.4% accuracy improvements over the sequential thinking model
trained directly on challenging tasks with RL. Further analysis reveals a clear
shift in the model's thinking behavior: at an early stage, it uses parallel
thinking as an exploration strategy, while in a later stage, it uses the same
capability for multi-perspective verification. Most significantly, we validate
parallel thinking as a mid-training exploration scaffold, where this
temporary exploratory phase unlocks a higher performance ceiling after RL,
yielding a 42.9% improvement over the baseline on AIME25. Our model, data, and
code will be open-source at https://github.com/zhengkid/Parallel-R1.
68c0d8e13912ed54cf543209
Tong Zheng
claimed_verified
2025-09-12T16:18:27.526Z
6623ea65b642e29cdf90a1b4
/avatars/e32e90574c1162b2be87ed78604e3e4d.svg
TongZheng
user
TongZheng1999
68c0d8e13912ed54cf54320a
Hongming Zhang
68c0d8e13912ed54cf54320b
Wenhao Yu
68c0d8e13912ed54cf54320c
Xiaoyang Wang
68c0d8e13912ed54cf54320d
Xinyu Yang
68c0d8e13912ed54cf54320e
Runpeng Dai
claimed_verified
2025-09-12T16:18:31.918Z
65037565da2d88e201f63b7a
/avatars/d1b6ce17236360e9583b8bb4cb87e506.svg
Runpeng Dai
user
Leo-Dai
68c0d8e13912ed54cf54320f
Rui Liu
claimed_verified
2025-09-12T16:18:24.186Z
6656bf615b203a05a1f0968c
/avatars/1ee0b0099c10dd76c8e3b7d312221b15.svg
Rui Liu
user
lr10260
68c0d8e13912ed54cf543210
Huiwen Bao
claimed_verified
2025-09-15T15:09:10.334Z
67c406f987a7f49a82811c49
/avatars/fa0cab9c296addcab8979904491d6dcd.svg
baohuiwen
user
baohuiwen
68c0d8e13912ed54cf543211
Chengsong Huang
claimed_verified
2025-09-12T16:18:36.269Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
user
ChengsongHuang
68c0d8e13912ed54cf543212
Heng Huang
68c0d8e13912ed54cf543213
Dong Yu
Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
Parallel thinking has emerged as a novel approach for enhancing the reasoning
capabilities of large language models (LLMs) by exploring multiple reasoning
paths concurrently. However, activating such capabilities through training
remains challenging, as existing methods predominantly rely on supervised
fine-tuning (SFT) over synthetic data, which encourages teacher-forced
imitation rather than exploration and generalization. Different from them, we
propose Parallel-R1, the first reinforcement learning (RL) framework
that enables parallel thinking behaviors for complex real-world reasoning
tasks. Our framework employs a progressive curriculum that explicitly addresses
the cold-start problem in training parallel thinking with RL. We first use SFT
on prompt-generated trajectories from easier tasks to instill the parallel
thinking ability, then transition to RL to explore and generalize this skill on
harder problems. Experiments on various math benchmarks, including MATH, AMC23,
and AIME, show that Parallel-R1 successfully instills parallel thinking,
leading to 8.4% accuracy improvements over the sequential thinking model
trained directly on challenging tasks with RL. Further analysis reveals a clear
shift in the model's thinking behavior: at an early stage, it uses parallel
thinking as an exploration strategy, while in a later stage, it uses the same
capability for multi-perspective verification. Most significantly, we validate
parallel thinking as a mid-training exploration scaffold, where this
temporary exploratory phase unlocks a higher performance ceiling after RL,
yielding a 42.9% improvement over the baseline on AIME25. Our model, data, and
code will be open-source at https://github.com/zhengkid/Parallel-R1.
Parallel-R1, a reinforcement learning framework, enhances large language models' reasoning capabilities by enabling parallel thinking through a progressive curriculum, leading to significant performance improvements on math benchmarks.
parallel thinking reinforcement learning progressive curriculum cold-start problem supervised fine-tuning prompt-generated trajectories sequential thinking multi-perspective verification mid-training exploration scaffold