RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
Recent progress in vision and language foundation models has significantly
advanced multimodal understanding, reasoning, and generation, inspiring a surge
of interest in extending such capabilities to embodied settings through
vision-language-action (VLA) models. Yet, most VLA models are still trained
with supervised fine-tuning (SFT), which struggles to generalize under
distribution shifts due to error accumulation. Reinforcement learning (RL)
offers a promising alternative by directly optimizing task performance through
interaction, but existing attempts remain fragmented and lack a unified
platform for fair and systematic comparison across model architectures and
algorithmic designs. To address this gap, we introduce RLinf-VLA, a unified and
efficient framework for scalable RL training of VLA models. The system adopts a
highly flexible resource allocation design that addresses the challenge of
integrating rendering, training, and inference in RL+VLA training. In
particular, for GPU-parallelized simulators, RLinf-VLA implements a novel
hybrid fine-grained pipeline allocation mode, achieving a 1.61x-1.88x speedup
in training. Through a unified interface, RLinf-VLA seamlessly supports diverse
VLA architectures (e.g., OpenVLA, OpenVLA-OFT), multiple RL algorithms (e.g.,
PPO, GRPO), and various simulators (e.g., ManiSkill, LIBERO). In simulation, a
unified model achieves 98.11\% across 130 LIBERO tasks and 97.66\% across 25
ManiSkill tasks. Beyond empirical performance, our study distills a set of best
practices for applying RL to VLA training and sheds light on emerging patterns
in this integration. Furthermore, we present preliminary deployment on a
real-world Franka robot, where RL-trained policies exhibit stronger
generalization than those trained with SFT. We envision RLinf-VLA as a
foundation to accelerate and standardize research on embodied intelligence.
68e764c97ae125f9582e6bbc
Hongzhi Zang
68e764c97ae125f9582e6bbd
Mingjie Wei
68e764c97ae125f9582e6bbe
Si Xu
68e764c97ae125f9582e6bbf
Yongji Wu
68e764c97ae125f9582e6bc0
Zhen Guo
68e764c97ae125f9582e6bc1
Yuanqing Wang
68e764c97ae125f9582e6bc2
Hao Lin
68e764c97ae125f9582e6bc3
Liangzhi Shi
68e764c97ae125f9582e6bc4
Yuqing Xie
68e764c97ae125f9582e6bc5
Zhexuan Xu
68e764c97ae125f9582e6bc6
Zhihao Liu
68e764c97ae125f9582e6bc7
Kang Chen
68e764c97ae125f9582e6bc8
Wenhao Tang
claimed_verified
2025-10-09T12:48:14.490Z
66a51845f9565635ebec3b55
/avatars/3cfff055e79da1bc43293b94b28ca210.svg
Wenhao Tang
user
tangwh0517
68e764c97ae125f9582e6bc9
Quanlu Zhang
68e764c97ae125f9582e6bca
Weinan Zhang
68e764c97ae125f9582e6bcb
Chao Yu
68e764c97ae125f9582e6bcc
Yu Wang
RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
Recent progress in vision and language foundation models has significantly
advanced multimodal understanding, reasoning, and generation, inspiring a surge
of interest in extending such capabilities to embodied settings through
vision-language-action (VLA) models. Yet, most VLA models are still trained
with supervised fine-tuning (SFT), which struggles to generalize under
distribution shifts due to error accumulation. Reinforcement learning (RL)
offers a promising alternative by directly optimizing task performance through
interaction, but existing attempts remain fragmented and lack a unified
platform for fair and systematic comparison across model architectures and
algorithmic designs. To address this gap, we introduce RLinf-VLA, a unified and
efficient framework for scalable RL training of VLA models. The system adopts a
highly flexible resource allocation design that addresses the challenge of
integrating rendering, training, and inference in RL+VLA training. In
particular, for GPU-parallelized simulators, RLinf-VLA implements a novel
hybrid fine-grained pipeline allocation mode, achieving a 1.61x-1.88x speedup
in training. Through a unified interface, RLinf-VLA seamlessly supports diverse
VLA architectures (e.g., OpenVLA, OpenVLA-OFT), multiple RL algorithms (e.g.,
PPO, GRPO), and various simulators (e.g., ManiSkill, LIBERO). In simulation, a
unified model achieves 98.11\% across 130 LIBERO tasks and 97.66\% across 25
ManiSkill tasks. Beyond empirical performance, our study distills a set of best
practices for applying RL to VLA training and sheds light on emerging patterns
in this integration. Furthermore, we present preliminary deployment on a
real-world Franka robot, where RL-trained policies exhibit stronger
generalization than those trained with SFT. We envision RLinf-VLA as a
foundation to accelerate and standardize research on embodied intelligence.
RLinf-VLA is a unified framework for scalable reinforcement learning training of vision-language-action models, offering improved performance and generalization compared to supervised fine-tuning.
vision-language-action models supervised fine-tuning reinforcement learning RLinf-VLA resource allocation GPU-parallelized simulators hybrid fine-grained pipeline allocation OpenVLA OpenVLA-OFT PPO GRPO ManiSkill LIBERO Franka robot