EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language
  Models for Vision-Driven Embodied Agents
Leveraging Multi-modal Large Language Models (MLLMs) to create embodied
agents offers a promising avenue for tackling real-world tasks. While
language-centric embodied agents have garnered substantial attention,
MLLM-based embodied agents remain underexplored due to the lack of
comprehensive evaluation frameworks. To bridge this gap, we introduce
EmbodiedBench, an extensive benchmark designed to evaluate vision-driven
embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing
tasks across four environments, ranging from high-level semantic tasks (e.g.,
household) to low-level tasks involving atomic actions (e.g., navigation and
manipulation); and (2) six meticulously curated subsets evaluating essential
agent capabilities like commonsense reasoning, complex instruction
understanding, spatial awareness, visual perception, and long-term planning.
Through extensive experiments, we evaluated 13 leading proprietary and
open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel
at high-level tasks but struggle with low-level manipulation, with the best
model, GPT-4o, scoring only 28.9% on average. EmbodiedBench provides a
multifaceted standardized evaluation platform that not only highlights existing
challenges but also offers valuable insights to advance MLLM-based embodied
agents. Our code is available at https://embodiedbench.github.io.
67aec4285b9801b819449b84
Rui Yang
claimed_verified
2025-02-18T09:34:05.913Z
64d45451c34a346181b130dd
/avatars/9bb8205b889337df5d321539c9b5d69d.svg
Rui Yang
user
Ray2333
67aec4285b9801b819449b85
Hanyang Chen
claimed_verified
2025-02-14T08:01:08.365Z
6700b1f93381f2db06857fb5
/avatars/c8b9ec7c00773c5a4055ba50de0c6b2f.svg
Hanyang Chen
user
Hanyang81
67aec4285b9801b819449b86
Junyu Zhang
claimed_verified
2025-02-18T09:34:03.521Z
6719bfd07c6e6c83a388aeae
https://cdn-avatars.huggingface.co/v1/production/uploads/6719bfd07c6e6c83a388aeae/jHxryk04dzHo23TX5F5sz.png
Junyu Zhang
user
jyzhang1208
67aec4285b9801b819449b87
Mark Zhao
67aec4285b9801b819449b88
Cheng Qian
claimed_verified
2025-04-23T08:38:49.914Z
665e121c6007027038fd4005
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sIVBJAGM-Kneq9KMf8aXb.png
Cheng Qian
user
chengq9
67aec4285b9801b819449b89
Kangrui Wang
admin_assigned
2025-02-14T12:47:36.061Z
66f2b8602ef2817ec3cb65f6
/avatars/c8c5b2706644fb45a75f13af99fa7ae9.svg
Kangrui Wang
user
JamesK2W
67aec4285b9801b819449b8a
Qineng Wang
admin_assigned
2025-02-14T12:46:06.895Z
640c3b28c5fa12d61a50cd92
/avatars/81556de3214c848b3c3e118f50fd2968.svg
Qineng Wang
user
Inevitablevalor
67aec4285b9801b819449b8b
Teja Venkat Koripella
67aec4285b9801b819449b8c
Marziyeh Movahedi
admin_assigned
2025-02-14T12:46:13.104Z
64e10c263209bf4194912319
/avatars/02f1a9e2ce333ff521d901cf83fcdff3.svg
Marziyeh Movahedi
user
Marzimv
67aec4285b9801b819449b8d
Manling Li
admin_assigned
2025-02-14T12:46:50.911Z
6746a140f2ca2162e3bcfe2b
/avatars/d9d8cfb5f112e6ed7f6152fc230135d3.svg
Manling Li
user
ManlingLi
67aec4285b9801b819449b8e
Heng Ji
67aec4285b9801b819449b8f
Huan Zhang
admin_assigned
2025-02-14T12:47:14.741Z
6719d581a6cad13741b8bc7f
https://cdn-avatars.huggingface.co/v1/production/uploads/6719d581a6cad13741b8bc7f/w4EttqfXRgWZJc6HpYOS9.jpeg
Huan Zhang
user
huanzhang12
67aec4285b9801b819449b90
Tong Zhang
EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language
  Models for Vision-Driven Embodied Agents
Leveraging Multi-modal Large Language Models (MLLMs) to create embodied
agents offers a promising avenue for tackling real-world tasks. While
language-centric embodied agents have garnered substantial attention,
MLLM-based embodied agents remain underexplored due to the lack of
comprehensive evaluation frameworks. To bridge this gap, we introduce
EmbodiedBench, an extensive benchmark designed to evaluate vision-driven
embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing
tasks across four environments, ranging from high-level semantic tasks (e.g.,
household) to low-level tasks involving atomic actions (e.g., navigation and
manipulation); and (2) six meticulously curated subsets evaluating essential
agent capabilities like commonsense reasoning, complex instruction
understanding, spatial awareness, visual perception, and long-term planning.
Through extensive experiments, we evaluated 13 leading proprietary and
open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel
at high-level tasks but struggle with low-level manipulation, with the best
model, GPT-4o, scoring only 28.9% on average. EmbodiedBench provides a
multifaceted standardized evaluation platform that not only highlights existing
challenges but also offers valuable insights to advance MLLM-based embodied
agents. Our code is available at https://embodiedbench.github.io.
EmbodiedBench is a benchmark comprising diverse tasks to evaluate MLLMs in embodied agents, revealing strengths and weaknesses across high-level and low-level capabilities.
Multi-modal Large Language Models (MLLMs) embodied agents EmbodiedBench vision-driven embodied agents commonsense reasoning complex instruction understanding spatial awareness visual perception long-term planning GPT-4o