InstructVLA: Vision-Language-Action Instruction Tuning from
  Understanding to Manipulation
To operate effectively in the real world, robots must integrate multimodal
reasoning with precise action generation. However, existing
vision-language-action (VLA) models often sacrifice one for the other, narrow
their abilities to task-specific manipulation data, and suffer catastrophic
forgetting of pre-trained vision-language capabilities. To bridge this gap, we
introduce InstructVLA, an end-to-end VLA model that preserves the flexible
reasoning of large vision-language models (VLMs) while delivering leading
manipulation performance. InstructVLA introduces a novel training paradigm,
Vision-Language-Action Instruction Tuning (VLA-IT), which employs multimodal
training with mixture-of-experts adaptation to jointly optimize textual
reasoning and action generation on both standard VLM corpora and a curated
650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves
30.5% improvement over SpatialVLA. To evaluate generalization, we introduce
SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and
high-level instruction understanding, where it outperforms a fine-tuned OpenVLA
by 92% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA
surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling
by leveraging textual reasoning to boost manipulation performance in both
simulated and real-world settings. These results demonstrate InstructVLA's
potential for bridging intuitive and steerable human-robot interaction with
efficient policy learning.
68917181f01a094725f8348d
Shuai Yang
claimed_verified
2025-08-06T19:22:13.248Z
64548f6c363bb3aaf9cba136
https://cdn-avatars.huggingface.co/v1/production/uploads/64548f6c363bb3aaf9cba136/HqJL9HQ5CWVOJCsHyuMOm.jpeg
Shuai Yang
user
ShuaiYang03
68917181f01a094725f8348e
Hao Li
68917181f01a094725f8348f
Yilun Chen
68917181f01a094725f83490
Bin Wang
68917181f01a094725f83491
Yang Tian
68917181f01a094725f83492
Tai Wang
68917181f01a094725f83493
Hanqing Wang
68917181f01a094725f83494
Feng Zhao
68917181f01a094725f83495
Yiyi Liao
68917181f01a094725f83496
Jiangmiao Pang
InstructVLA: Vision-Language-Action Instruction Tuning from
  Understanding to Manipulation
To operate effectively in the real world, robots must integrate multimodal
reasoning with precise action generation. However, existing
vision-language-action (VLA) models often sacrifice one for the other, narrow
their abilities to task-specific manipulation data, and suffer catastrophic
forgetting of pre-trained vision-language capabilities. To bridge this gap, we
introduce InstructVLA, an end-to-end VLA model that preserves the flexible
reasoning of large vision-language models (VLMs) while delivering leading
manipulation performance. InstructVLA introduces a novel training paradigm,
Vision-Language-Action Instruction Tuning (VLA-IT), which employs multimodal
training with mixture-of-experts adaptation to jointly optimize textual
reasoning and action generation on both standard VLM corpora and a curated
650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves
30.5% improvement over SpatialVLA. To evaluate generalization, we introduce
SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and
high-level instruction understanding, where it outperforms a fine-tuned OpenVLA
by 92% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA
surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling
by leveraging textual reasoning to boost manipulation performance in both
simulated and real-world settings. These results demonstrate InstructVLA's
potential for bridging intuitive and steerable human-robot interaction with
efficient policy learning.
InstructVLA is an end-to-end vision-language-action model that enhances manipulation performance while preserving vision-language reasoning through multimodal training and mixture-of-experts adaptation.
vision-language-action (VLA) models multimodal reasoning precise action generation catastrophic forgetting large vision-language models (VLMs) Vision-Language-Action Instruction Tuning (VLA-IT) mixture-of-experts adaptation SimplerEnv tasks SimplerEnv-Instruct OpenVLA GPT-4o multimodal tasks inference-time scaling policy learning