From Intention to Execution: Probing the Generalization Boundaries of
  Vision-Language-Action Models
One promise that Vision-Language-Action (VLA) models hold over traditional
imitation learning for robotics is to leverage the broad generalization
capabilities of large Vision-Language Models (VLMs) to produce versatile,
"generalist" robot policies. However, current evaluations of VLAs remain
insufficient. Traditional imitation learning benchmarks are unsuitable due to
the lack of language instructions. Emerging benchmarks for VLAs that
incorporate language often come with limited evaluation tasks and do not intend
to investigate how much VLM pretraining truly contributes to the generalization
capabilities of the downstream robotic policy. Meanwhile, much research relies
on real-world robot setups designed in isolation by different institutions,
which creates a barrier for reproducibility and accessibility. To address this
gap, we introduce a unified probing suite of 50 simulation-based tasks across
10 subcategories spanning language instruction, vision, and objects. We
systematically evaluate several state-of-the-art VLA architectures on this
suite to understand their generalization capability. Our results show that
while VLM backbones endow VLAs with robust perceptual understanding and high
level planning, which we refer to as good intentions, this does not reliably
translate into precise motor execution: when faced with out-of-distribution
observations, policies often exhibit coherent intentions, but falter in action
execution. Moreover, finetuning on action data can erode the original VLM's
generalist reasoning abilities. We release our task suite and evaluation code
to serve as a standardized benchmark for future VLAs and to drive research on
closing the perception-to-action gap. More information, including the source
code, can be found at https://ai4ce.github.io/INT-ACT/
684c759d3b733ba3336872d3
Irving Fang
684c759d3b733ba3336872d4
Juexiao Zhang
claimed_verified
2025-06-14T06:24:20.800Z
65119b03d6e446efb3a5b4ce
https://cdn-avatars.huggingface.co/v1/production/uploads/65119b03d6e446efb3a5b4ce/axKQipd3gPsoO0I3FNL0O.jpeg
Juexiao Zhang
user
juexzz
684c759d3b733ba3336872d5
Shengbang Tong
684c759d3b733ba3336872d6
Chen Feng
From Intention to Execution: Probing the Generalization Boundaries of
  Vision-Language-Action Models
One promise that Vision-Language-Action (VLA) models hold over traditional
imitation learning for robotics is to leverage the broad generalization
capabilities of large Vision-Language Models (VLMs) to produce versatile,
"generalist" robot policies. However, current evaluations of VLAs remain
insufficient. Traditional imitation learning benchmarks are unsuitable due to
the lack of language instructions. Emerging benchmarks for VLAs that
incorporate language often come with limited evaluation tasks and do not intend
to investigate how much VLM pretraining truly contributes to the generalization
capabilities of the downstream robotic policy. Meanwhile, much research relies
on real-world robot setups designed in isolation by different institutions,
which creates a barrier for reproducibility and accessibility. To address this
gap, we introduce a unified probing suite of 50 simulation-based tasks across
10 subcategories spanning language instruction, vision, and objects. We
systematically evaluate several state-of-the-art VLA architectures on this
suite to understand their generalization capability. Our results show that
while VLM backbones endow VLAs with robust perceptual understanding and high
level planning, which we refer to as good intentions, this does not reliably
translate into precise motor execution: when faced with out-of-distribution
observations, policies often exhibit coherent intentions, but falter in action
execution. Moreover, finetuning on action data can erode the original VLM's
generalist reasoning abilities. We release our task suite and evaluation code
to serve as a standardized benchmark for future VLAs and to drive research on
closing the perception-to-action gap. More information, including the source
code, can be found at https://ai4ce.github.io/INT-ACT/
A unified benchmark suite evaluates Vision-Language-Action models' generalization and motor execution capabilities, highlighting the disparity between perceptual understanding and precise action execution.
Vision-Language-Action (VLA) models Vision-Language Models (VLMs) imitation learning language instruction generalization capabilities simulation-based tasks perceptual understanding high level planning motor execution out-of-distribution observations finetuning generalist reasoning abilities