VAPO: Efficient and Reliable Reinforcement Learning for Advanced
  Reasoning Tasks
We present VAPO, Value-based Augmented Proximal Policy Optimization framework
for reasoning models., a novel framework tailored for reasoning models within
the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the
Qwen 32B pre-trained model, attains a state-of-the-art score of
60.4. In direct comparison under identical experimental settings,
VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B
and DAPO by more than 10 points. The training process of VAPO stands out for
its stability and efficiency. It reaches state-of-the-art performance within a
mere 5,000 steps. Moreover, across multiple independent runs, no training
crashes occur, underscoring its reliability. This research delves into long
chain-of-thought (long-CoT) reasoning using a value-based reinforcement
learning framework. We pinpoint three key challenges that plague value-based
methods: value model bias, the presence of heterogeneous sequence lengths, and
the sparsity of reward signals. Through systematic design, VAPO offers an
integrated solution that effectively alleviates these challenges, enabling
enhanced performance in long-CoT reasoning tasks.
67f503bdcb45555af74243e3
YuYue
67f503bdcb45555af74243e4
Yufeng Yuan
67f503bdcb45555af74243e5
Qiying Yu
67f503bdcb45555af74243e6
Xiaochen Zuo
67f503bdcb45555af74243e7
Ruofei Zhu
67f503bdcb45555af74243e8
Wenyuan Xu
67f503bdcb45555af74243e9
Jiaze Chen
67f503bdcb45555af74243ea
Chengyi Wang
67f503bdcb45555af74243eb
TianTian Fan
67f503bdcb45555af74243ec
Zhengyin Du
67f503bdcb45555af74243ed
Xiangpeng Wei
67f503bdcb45555af74243ee
Gaohong Liu
67f503bdcb45555af74243ef
Juncai Liu
67f503bdcb45555af74243f0
Lingjun Liu
67f503bdcb45555af74243f1
Haibin Lin
67f503bdcb45555af74243f2
Zhiqi Lin
67f503bdcb45555af74243f3
Bole Ma
67f503bdcb45555af74243f4
Chi Zhang
67f503bdcb45555af74243f5
Mofan Zhang
67f503bdcb45555af74243f6
Wang Zhang
67f503bdcb45555af74243f7
Hang Zhu
67f503bdcb45555af74243f8
Ru Zhang
67f503bdcb45555af74243f9
Xin Liu
67f503bdcb45555af74243fa
Mingxuan Wang
67f503bdcb45555af74243fb
Yonghui Wu
67f503bdcb45555af74243fc
Lin Yan
VAPO: Efficient and Reliable Reinforcement Learning for Advanced
  Reasoning Tasks
We present VAPO, Value-based Augmented Proximal Policy Optimization framework
for reasoning models., a novel framework tailored for reasoning models within
the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the
Qwen 32B pre-trained model, attains a state-of-the-art score of
60.4. In direct comparison under identical experimental settings,
VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B
and DAPO by more than 10 points. The training process of VAPO stands out for
its stability and efficiency. It reaches state-of-the-art performance within a
mere 5,000 steps. Moreover, across multiple independent runs, no training
crashes occur, underscoring its reliability. This research delves into long
chain-of-thought (long-CoT) reasoning using a value-based reinforcement
learning framework. We pinpoint three key challenges that plague value-based
methods: value model bias, the presence of heterogeneous sequence lengths, and
the sparsity of reward signals. Through systematic design, VAPO offers an
integrated solution that effectively alleviates these challenges, enabling
enhanced performance in long-CoT reasoning tasks.
VAPO, a value-based augmented proximal policy optimization framework, achieves state-of-the-art performance on reasoning tasks using a value-based reinforcement learning approach.
Value-based Augmented Proximal Policy Optimization value-based paradigm Qwen 32B long chain-of-thought reasoning long-CoT reasoning value model bias heterogeneous sequence lengths sparsity of reward signals reinforcement learning