Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn
  Dialogue
The ultimate goal of embodied agents is to create collaborators that can
interact with humans, not mere executors that passively follow instructions.
This requires agents to communicate, coordinate, and adapt their actions based
on human feedback. Recently, advances in VLAs have offered a path toward this
goal. However, most current VLA-based embodied agents operate in a one-way
mode: they receive an instruction and execute it without feedback. This
approach fails in real-world scenarios where instructions are often ambiguous.
In this paper, we address this problem with the Ask-to-Clarify framework. Our
framework first resolves ambiguous instructions by asking questions in a
multi-turn dialogue. Then it generates low-level actions end-to-end.
Specifically, the Ask-to-Clarify framework consists of two components, one VLM
for collaboration and one diffusion for action. We also introduce a connection
module that generates conditions for the diffusion based on the output of the
VLM. This module adjusts the observation by instructions to create reliable
conditions. We train our framework with a two-stage knowledge-insulation
strategy. First, we fine-tune the collaboration component using
ambiguity-solving dialogue data to handle ambiguity. Then, we integrate the
action component while freezing the collaboration one. This preserves the
interaction abilities while fine-tuning the diffusion to generate actions. The
training strategy guarantees our framework can first ask questions, then
generate actions. During inference, a signal detector functions as a router
that helps our framework switch between asking questions and taking actions. We
evaluate the Ask-to-Clarify framework in 8 real-world tasks, where it
outperforms existing state-of-the-art VLAs. The results suggest that our
proposed framework, along with the training strategy, provides a path toward
collaborative embodied agents.
68ccbd313df9ac65e93dc65a
Xingyao Lin
claimed_verified
2025-09-19T06:48:30.289Z
66618028781f4604bab56fd2
/avatars/2f226ba9bfa0afcec9b5f65714f33137.svg
Leo
user
leolin9248
68ccbd313df9ac65e93dc65b
Xinghao Zhu
68ccbd313df9ac65e93dc65c
Tianyi Lu
68ccbd313df9ac65e93dc65d
Sicheng Xie
68ccbd313df9ac65e93dc65e
Hui Zhang
68ccbd313df9ac65e93dc65f
Xipeng Qiu
68ccbd313df9ac65e93dc660
Zuxuan Wu
68ccbd313df9ac65e93dc661
Yu-Gang Jiang
Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn
  Dialogue
The ultimate goal of embodied agents is to create collaborators that can
interact with humans, not mere executors that passively follow instructions.
This requires agents to communicate, coordinate, and adapt their actions based
on human feedback. Recently, advances in VLAs have offered a path toward this
goal. However, most current VLA-based embodied agents operate in a one-way
mode: they receive an instruction and execute it without feedback. This
approach fails in real-world scenarios where instructions are often ambiguous.
In this paper, we address this problem with the Ask-to-Clarify framework. Our
framework first resolves ambiguous instructions by asking questions in a
multi-turn dialogue. Then it generates low-level actions end-to-end.
Specifically, the Ask-to-Clarify framework consists of two components, one VLM
for collaboration and one diffusion for action. We also introduce a connection
module that generates conditions for the diffusion based on the output of the
VLM. This module adjusts the observation by instructions to create reliable
conditions. We train our framework with a two-stage knowledge-insulation
strategy. First, we fine-tune the collaboration component using
ambiguity-solving dialogue data to handle ambiguity. Then, we integrate the
action component while freezing the collaboration one. This preserves the
interaction abilities while fine-tuning the diffusion to generate actions. The
training strategy guarantees our framework can first ask questions, then
generate actions. During inference, a signal detector functions as a router
that helps our framework switch between asking questions and taking actions. We
evaluate the Ask-to-Clarify framework in 8 real-world tasks, where it
outperforms existing state-of-the-art VLAs. The results suggest that our
proposed framework, along with the training strategy, provides a path toward
collaborative embodied agents.
The Ask-to-Clarify framework uses a VLM for collaboration and a diffusion model for action generation, enabling embodied agents to handle ambiguous instructions through multi-turn dialogue and outperform existing VLAs in real-world tasks.
VLA embodied agents Ask-to-Clarify framework VLM diffusion connection module two-stage knowledge-insulation strategy signal detector