OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and
  Embodiment-aware Reasoning
Recent advances in multimodal large language models (MLLMs) have opened new
opportunities for embodied intelligence, enabling multimodal understanding,
reasoning, and interaction, as well as continuous spatial decision-making.
Nevertheless, current MLLM-based embodied systems face two critical
limitations. First, Geometric Adaptability Gap: models trained solely on 2D
inputs or with hard-coded 3D geometry injection suffer from either insufficient
spatial information or restricted 2D generalization, leading to poor
adaptability across tasks with diverse spatial demands. Second, Embodiment
Constraint Gap: prior work often neglects the physical constraints and
capacities of real robots, resulting in task plans that are theoretically valid
but practically infeasible.To address these gaps, we introduce OmniEVA -- an
embodied versatile planner that enables advanced embodied reasoning and task
planning through two pivotal innovations: (1) a Task-Adaptive 3D Grounding
mechanism, which introduces a gated router to perform explicit selective
regulation of 3D fusion based on contextual requirements, enabling
context-aware 3D grounding for diverse embodied tasks. (2) an Embodiment-Aware
Reasoning framework that jointly incorporates task goals and embodiment
constraints into the reasoning loop, resulting in planning decisions that are
both goal-directed and executable. Extensive experimental results demonstrate
that OmniEVA not only achieves state-of-the-art general embodied reasoning
performance, but also exhibits a strong ability across a wide range of
downstream scenarios. Evaluations of a suite of proposed embodied benchmarks,
including both primitive and composite tasks, confirm its robust and versatile
planning capabilities. Project page: https://omnieva.github.io
68c382b8fc1747b912403a1c
Yuecheng Liu
68c382b8fc1747b912403a1d
Dafeng Chi
68c382b8fc1747b912403a1e
Shiguang Wu
68c382b8fc1747b912403a1f
Zhanguang Zhang
68c382b8fc1747b912403a20
Yuzheng Zhuang
68c382b8fc1747b912403a21
Bowen Yang
68c382b8fc1747b912403a22
He Zhu
68c382b8fc1747b912403a23
Lingfeng Zhang
68c382b8fc1747b912403a24
Pengwei Xie
68c382b8fc1747b912403a25
David Gamaliel Arcos Bravo
68c382b8fc1747b912403a26
Yingxue Zhang
68c382b8fc1747b912403a27
Jianye Hao
68c382b8fc1747b912403a28
Xingyue Quan
OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and
  Embodiment-aware Reasoning
Recent advances in multimodal large language models (MLLMs) have opened new
opportunities for embodied intelligence, enabling multimodal understanding,
reasoning, and interaction, as well as continuous spatial decision-making.
Nevertheless, current MLLM-based embodied systems face two critical
limitations. First, Geometric Adaptability Gap: models trained solely on 2D
inputs or with hard-coded 3D geometry injection suffer from either insufficient
spatial information or restricted 2D generalization, leading to poor
adaptability across tasks with diverse spatial demands. Second, Embodiment
Constraint Gap: prior work often neglects the physical constraints and
capacities of real robots, resulting in task plans that are theoretically valid
but practically infeasible.To address these gaps, we introduce OmniEVA -- an
embodied versatile planner that enables advanced embodied reasoning and task
planning through two pivotal innovations: (1) a Task-Adaptive 3D Grounding
mechanism, which introduces a gated router to perform explicit selective
regulation of 3D fusion based on contextual requirements, enabling
context-aware 3D grounding for diverse embodied tasks. (2) an Embodiment-Aware
Reasoning framework that jointly incorporates task goals and embodiment
constraints into the reasoning loop, resulting in planning decisions that are
both goal-directed and executable. Extensive experimental results demonstrate
that OmniEVA not only achieves state-of-the-art general embodied reasoning
performance, but also exhibits a strong ability across a wide range of
downstream scenarios. Evaluations of a suite of proposed embodied benchmarks,
including both primitive and composite tasks, confirm its robust and versatile
planning capabilities. Project page: https://omnieva.github.io
OmniEVA addresses spatial and embodiment gaps in multimodal large language models for embodied intelligence through a task-adaptive 3D grounding mechanism and an embodiment-aware reasoning framework, achieving state-of-the-art performance across diverse tasks.
multimodal large language models embodied intelligence multimodal understanding reasoning interaction spatial decision-making Geometric Adaptability Gap Embodiment Constraint Gap Task-Adaptive 3D Grounding gated router 3D fusion context-aware 3D grounding Embodiment-Aware Reasoning goal-directed executable embodied benchmarks primitive tasks composite tasks