VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing,
  Speaking, and Acting
Current Vision-Language-Action (VLA) models are often constrained by a rigid,
static interaction paradigm, which lacks the ability to see, hear, speak, and
act concurrently as well as handle real-time user interruptions dynamically.
This hinders seamless embodied collaboration, resulting in an inflexible and
unresponsive user experience. To address these limitations, we introduce
VITA-E, a novel embodied interaction framework designed for both behavioral
concurrency and nearly real-time interruption. The core of our approach is a
dual-model architecture where two parallel VLA instances operate as an ``Active
Model'' and a ``Standby Model'', allowing the embodied agent to observe its
environment, listen to user speech, provide verbal responses, and execute
actions, all concurrently and interruptibly, mimicking human-like multitasking
capabilities. We further propose a ``model-as-controller'' paradigm, where we
fine-tune the VLM to generate special tokens that serve as direct system-level
commands, coupling the model's reasoning with the system's behavior.
Experiments conducted on a physical humanoid platform demonstrate that VITA-E
can reliably handle complex interactive scenarios. Our framework is compatible
with various dual-system VLA models, achieving an extremely high success rate
on emergency stops and speech interruptions while also successfully performing
concurrent speech and action. This represents a significant step towards more
natural and capable embodied assistants.
69002d2622d452aac6dd42d4
Xiaoyu Liu
69002d2622d452aac6dd42d5
Chaoyou Fu
69002d2622d452aac6dd42d6
Chi Yan
69002d2622d452aac6dd42d7
Chu Wu
69002d2622d452aac6dd42d8
Haihan Gao
69002d2622d452aac6dd42d9
Yi-Fan Zhang
69002d2622d452aac6dd42da
Shaoqi Dong
69002d2622d452aac6dd42db
Cheng Qian
69002d2622d452aac6dd42dc
Bin Luo
69002d2622d452aac6dd42dd
Xiuyong Yang
69002d2622d452aac6dd42de
Guanwu Li
69002d2622d452aac6dd42df
Yusheng Cai
69002d2622d452aac6dd42e0
Yunhang Shen
69002d2622d452aac6dd42e1
Deqiang Jiang
69002d2622d452aac6dd42e2
Haoyu Cao
69002d2622d452aac6dd42e3
Xing Sun
claimed_verified
2025-10-28T15:36:42.216Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
user
tedsun
69002d2622d452aac6dd42e4
Caifeng Shan
69002d2622d452aac6dd42e5
Ran He
VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing,
  Speaking, and Acting
Current Vision-Language-Action (VLA) models are often constrained by a rigid,
static interaction paradigm, which lacks the ability to see, hear, speak, and
act concurrently as well as handle real-time user interruptions dynamically.
This hinders seamless embodied collaboration, resulting in an inflexible and
unresponsive user experience. To address these limitations, we introduce
VITA-E, a novel embodied interaction framework designed for both behavioral
concurrency and nearly real-time interruption. The core of our approach is a
dual-model architecture where two parallel VLA instances operate as an ``Active
Model'' and a ``Standby Model'', allowing the embodied agent to observe its
environment, listen to user speech, provide verbal responses, and execute
actions, all concurrently and interruptibly, mimicking human-like multitasking
capabilities. We further propose a ``model-as-controller'' paradigm, where we
fine-tune the VLM to generate special tokens that serve as direct system-level
commands, coupling the model's reasoning with the system's behavior.
Experiments conducted on a physical humanoid platform demonstrate that VITA-E
can reliably handle complex interactive scenarios. Our framework is compatible
with various dual-system VLA models, achieving an extremely high success rate
on emergency stops and speech interruptions while also successfully performing
concurrent speech and action. This represents a significant step towards more
natural and capable embodied assistants.
VITA-E, a dual-model embodied interaction framework, enables concurrent and interruptible vision-language-action capabilities, enhancing real-time user interaction.
embodied interaction framework dual-model architecture Active Model Standby Model model-as-controller VLM special tokens system-level commands emergency stops speech interruptions concurrent speech and action