Do What? Teaching Vision-Language-Action Models to Reject the Impossible
Recently, Vision-Language-Action (VLA) models have demonstrated strong
performance on a range of robotic tasks. These models rely on multimodal
inputs, with language instructions playing a crucial role -- not only in
predicting actions, but also in robustly interpreting user intent, even when
the requests are impossible to fulfill. In this work, we investigate how VLAs
can recognize, interpret, and respond to false-premise instructions: natural
language commands that reference objects or conditions absent from the
environment. We propose Instruct-Verify-and-Act (IVA), a unified framework that
(i) detects when an instruction cannot be executed due to a false premise, (ii)
engages in language-based clarification or correction, and (iii) grounds
plausible alternatives in perception and action. Towards this end, we construct
a large-scale instruction tuning setup with structured language prompts and
train a VLA model capable of handling both accurate and erroneous requests. Our
approach leverages a contextually augmented, semi-synthetic dataset containing
paired positive and false-premise instructions, enabling robust detection and
natural language correction. Our experiments show that IVA improves false
premise detection accuracy by 97.56% over baselines, while increasing
successful responses in false-premise scenarios by 50.78%.
68abe15486b21a0e2e358ad7
Wen-Han Hsieh
admin_assigned
2025-08-25T20:33:50.102Z
67aefef1bae0204268556790
/avatars/cc30835fca7855926360a00c012b88bc.svg
Wen Han Hsieh
user
wen-han
68abe15486b21a0e2e358ad8
Elvis Hsieh
68abe15486b21a0e2e358ad9
Dantong Niu
68abe15486b21a0e2e358ada
Trevor Darrell
68abe15486b21a0e2e358adb
Roei Herzig
68abe15486b21a0e2e358adc
David M. Chan
Do What? Teaching Vision-Language-Action Models to Reject the Impossible
Recently, Vision-Language-Action (VLA) models have demonstrated strong
performance on a range of robotic tasks. These models rely on multimodal
inputs, with language instructions playing a crucial role -- not only in
predicting actions, but also in robustly interpreting user intent, even when
the requests are impossible to fulfill. In this work, we investigate how VLAs
can recognize, interpret, and respond to false-premise instructions: natural
language commands that reference objects or conditions absent from the
environment. We propose Instruct-Verify-and-Act (IVA), a unified framework that
(i) detects when an instruction cannot be executed due to a false premise, (ii)
engages in language-based clarification or correction, and (iii) grounds
plausible alternatives in perception and action. Towards this end, we construct
a large-scale instruction tuning setup with structured language prompts and
train a VLA model capable of handling both accurate and erroneous requests. Our
approach leverages a contextually augmented, semi-synthetic dataset containing
paired positive and false-premise instructions, enabling robust detection and
natural language correction. Our experiments show that IVA improves false
premise detection accuracy by 97.56% over baselines, while increasing
successful responses in false-premise scenarios by 50.78%.
A unified framework, Instruct-Verify-and-Act (IVA), enhances Vision-Language-Action (VLA) models to detect and respond to false-premise instructions by leveraging contextually augmented datasets.
Vision-Language-Action models multimodal inputs language instructions false-premise instructions Instruct-Verify-and-Act contextually augmented semi-synthetic dataset